2024-06-14T23:49:36.513 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-14T23:49:36.513 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057 --owner scheduled_yuriw@teuthology --name yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi --verbose --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} -- /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/orig.config.yaml 2024-06-14T23:49:36.563 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-14T23:49:36.651 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057 branch: wip-yuri5-testing-2024-06-08-0902-reef description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7756057' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-06-08-0902-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-reef-drop-centos8 sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 302 sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: wip-reef-drop-centos8 suite_path: /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc targets: smithi063.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM/tWcxtI39XcfF9wLB4weKnA8NhwADc5iqueg6k2CebPVBv9KyuXnRZKGmuTqHpD837Ieln7VzjPLT8yWV2jg= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDctRE0YsEUzpmLRMiV8b73O8qU9BnTvitXzWcqaaWnuJQqu+Oc7PSmBagsBDSzNe9vss2qSftFhTbHPKXA03cE= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - ceph-fuse: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-14_18:29:32 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-14T23:49:36.652 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa; will attempt to use it 2024-06-14T23:49:36.652 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks 2024-06-14T23:49:36.653 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-14T23:49:36.654 INFO:teuthology.task.internal:Checking packages... 2024-06-14T23:49:36.674 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6f4b09560ed399c3f139402a02f389c2fc8bb2cc' 2024-06-14T23:49:36.674 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:49:36.675 INFO:teuthology.packaging:ref: None 2024-06-14T23:49:36.675 INFO:teuthology.packaging:tag: None 2024-06-14T23:49:36.675 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-06-08-0902-reef 2024-06-14T23:49:36.675 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:49:36.675 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri5-testing-2024-06-08-0902-reef 2024-06-14T23:49:36.886 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1443.g6f4b0956 2024-06-14T23:49:36.887 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-14T23:49:36.893 INFO:teuthology.task.internal:no buildpackages task found 2024-06-14T23:49:36.893 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-14T23:49:36.942 INFO:teuthology.task.internal:Saving configuration 2024-06-14T23:49:36.960 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-14T23:49:36.967 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-14T23:49:36.992 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi063.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-14 23:44:26.520054', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIM/tWcxtI39XcfF9wLB4weKnA8NhwADc5iqueg6k2CebPVBv9KyuXnRZKGmuTqHpD837Ieln7VzjPLT8yWV2jg='} 2024-06-14T23:49:37.016 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-14 23:44:26.521439', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDctRE0YsEUzpmLRMiV8b73O8qU9BnTvitXzWcqaaWnuJQqu+Oc7PSmBagsBDSzNe9vss2qSftFhTbHPKXA03cE='} 2024-06-14T23:49:37.017 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-14T23:49:37.023 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-14T23:49:37.023 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-14T23:49:37.023 INFO:teuthology.run_tasks:Running task console_log... 2024-06-14T23:49:37.103 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f55bf3cbbe0>, signals=[15]) 2024-06-14T23:49:37.103 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-14T23:49:37.109 INFO:teuthology.task.internal:Opening connections... 2024-06-14T23:49:37.109 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2024-06-14T23:49:37.110 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:49:37.185 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-06-14T23:49:37.186 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:49:37.257 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-14T23:49:37.263 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2024-06-14T23:49:37.281 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2024-06-14T23:49:37.281 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="CentOS Stream" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="9" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:ID="centos" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE="rhel fedora" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="9" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:PLATFORM_ID="platform:el9" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-14T23:49:37.339 INFO:teuthology.orchestra.run.smithi063.stdout:ANSI_COLOR="0;31" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:LOGO="fedora-logo-icon" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://centos.org/" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-14T23:49:37.340 INFO:teuthology.orchestra.run.smithi063.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-14T23:49:37.341 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2024-06-14T23:49:37.363 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-06-14T23:49:37.381 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-06-14T23:49:37.381 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-14T23:49:37.438 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-14T23:49:37.439 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-14T23:49:37.439 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-06-14T23:49:37.463 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-14T23:49:37.472 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-14T23:49:37.492 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-14T23:49:37.492 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2024-06-14T23:49:37.494 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-06-14T23:49:37.510 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-14T23:49:37.516 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-14T23:49:37.516 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2024-06-14T23:49:37.552 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-06-14T23:49:37.580 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-14T23:49:37.636 INFO:teuthology.run_tasks:Running task kernel... 2024-06-14T23:49:37.651 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-14T23:49:37.651 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-14T23:49:37.651 DEBUG:teuthology.orchestra.run.smithi063:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-14T23:49:37.651 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-14T23:49:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:49:37.668 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-06-14T23:49:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:49:37.669 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-06-14T23:49:37.725 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-437.el9.x86_64 2024-06-14T23:49:37.725 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-437.el9.x86_64 2024-06-14T23:49:37.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-06-14T23:49:37.726 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-437.el9.x86_64 2024-06-14T23:49:37.726 INFO:teuthology.task.kernel:Running kernel on smithi063: 5.14.0-437.el9.x86_64 2024-06-14T23:49:37.726 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum install -y kernel 2024-06-14T23:49:41.192 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-06-14T23:49:41.504 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.1 MB 00:02 2024-06-14T23:49:45.634 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 20 MB 00:03 2024-06-14T23:49:51.670 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 2.2 MB/s | 20 MB 00:09 2024-06-14T23:49:52.168 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.3 MB 00:02 2024-06-14T23:49:53.619 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 80 kB/s | 17 kB 00:00 2024-06-14T23:49:54.548 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-06-14T23:49:56.062 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.3 MB 00:00 2024-06-14T23:49:57.887 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 17 kB 00:00 2024-06-14T23:49:58.783 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-06-14T23:50:00.020 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-14T23:50:02.160 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-14T23:50:02.160 INFO:teuthology.orchestra.run.smithi063.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-14T23:50:02.258 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-14T23:50:02.264 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:02.265 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 81 M 2024-06-14T23:50:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 124 M 2024-06-14T23:50:02.266 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:50:02.555 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 4.4 MB/s | 989 kB 00:00 2024-06-14T23:50:03.072 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-06-14T23:50:03.722 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 25 MB/s | 29 MB 00:01 2024-06-14T23:50:04.056 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 20 MB/s | 35 MB 00:01 2024-06-14T23:50:04.056 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:50:04.056 INFO:teuthology.orchestra.run.smithi063.stdout:Total 45 MB/s | 81 MB 00:01 2024-06-14T23:50:04.080 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-06-14T23:50:04.297 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jun 2024 11:50:03 PM UTC. 2024-06-14T23:50:04.785 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:50:05.179 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:50:05.179 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:50:06.086 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-14T23:50:06.086 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-14T23:50:06.146 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:50:06.147 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:50:06.174 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:50:06.183 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-06-14T23:50:06.184 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:50:06.185 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 81 M 2024-06-14T23:50:06.185 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 124 M 2024-06-14T23:50:06.185 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:50:06.457 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 4.7 MB/s | 989 kB 00:00 2024-06-14T23:50:08.708 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:50:09.376 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-14T23:50:09.464 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:50:10.848 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-06-14T23:50:12.270 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:50:12.404 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:50:14.681 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 3.5 MB/s | 29 MB 00:08 2024-06-14T23:50:16.439 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 3.4 MB/s | 35 MB 00:10 2024-06-14T23:50:16.439 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:50:16.439 INFO:teuthology.orchestra.run.smithi136.stdout:Total 7.9 MB/s | 81 MB 00:10 2024-06-14T23:50:17.145 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:50:17.485 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:50:17.486 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:50:17.504 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:50:17.544 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:18.442 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:50:18.442 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:50:20.838 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:50:21.448 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-14T23:50:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:50:22.665 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:24.200 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:50:24.348 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:50:29.090 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:50:29.137 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:33.883 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:53.925 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:53.967 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:54.398 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:54.399 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-14T23:50:54.399 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:50:54.399 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.209 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.210 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:55.210 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:50:55.594 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T23:50:55.680 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.680 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T23:50:55.680 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-14T23:50:55.680 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-14T23:50:55.680 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-14T23:50:55.681 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-14T23:50:55.681 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2024-06-14T23:50:55.681 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T23:50:55.735 INFO:teuthology.orchestra.run.smithi063.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T23:50:55.736 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T23:50:55.736 DEBUG:teuthology.orchestra.run.smithi063:> sudo rpm -qi grub2-tools 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Name : grub2-tools 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Epoch : 1 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Version : 2.06 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Release : 68.el9 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: x86_64 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Group : Unspecified 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Size : 8264254 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:License : GPLv3+ 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-14T23:50:55.767 INFO:teuthology.orchestra.run.smithi063.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:Packager : builder@centos.org 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:Vendor : CentOS 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:Summary : Support tools for GRUB. 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:Description : 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:hardware devices. 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:50:55.768 INFO:teuthology.orchestra.run.smithi063.stdout:This subpackage provides tools for support of all platforms. 2024-06-14T23:50:55.769 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-14T23:50:55.769 INFO:teuthology.task.kernel:Updating grub on smithi063 to boot 5.14.0-457.el9.x86_64 2024-06-14T23:50:55.769 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-14T23:50:56.389 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2024-06-14T23:50:57.913 INFO:teuthology.orchestra.run.smithi063.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-14T23:50:57.945 INFO:teuthology.orchestra.run.smithi063.stderr:done 2024-06-14T23:50:57.948 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2024-06-14T23:50:58.012 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.GpvgeYT9NQ 2024-06-14T23:50:58.012 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.GpvgeYT9NQ 2024-06-14T23:50:58.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.GpvgeYT9NQ 2024-06-14T23:50:58.220 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.GpvgeYT9NQ is 6KB 2024-06-14T23:50:58.269 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.GpvgeYT9NQ 2024-06-14T23:50:58.284 DEBUG:teuthology.orchestra.run.smithi063:> sudo /bin/ls /boot/loader/entries || true 2024-06-14T23:50:58.345 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-14T23:50:58.345 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-14T23:50:58.346 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-14T23:50:58.346 INFO:teuthology.orchestra.run.smithi063.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-14T23:50:58.347 DEBUG:teuthology.orchestra.run.smithi063:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-14T23:50:58.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2024-06-14T23:51:03.220 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:51:03.260 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:51:03.635 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:51:03.635 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-14T23:51:03.635 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-14T23:51:03.635 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-14T23:51:04.462 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-14T23:51:04.463 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-14T23:51:04.463 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:51:04.463 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:51:04.878 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T23:51:04.986 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T23:51:04.986 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T23:51:04.987 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-14T23:51:04.987 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-14T23:51:04.987 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-14T23:51:04.987 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-14T23:51:04.987 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-06-14T23:51:04.987 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-06-14T23:51:05.040 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-14T23:51:05.040 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-14T23:51:05.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-06-14T23:51:05.070 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:51:05.071 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-06-14T23:51:05.072 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-14T23:51:05.072 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-457.el9.x86_64 2024-06-14T23:51:05.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-14T23:51:05.681 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-06-14T23:51:07.179 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-14T23:51:07.228 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-06-14T23:51:07.233 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-06-14T23:51:07.250 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.vWxpnTbEiQ 2024-06-14T23:51:07.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.vWxpnTbEiQ 2024-06-14T23:51:07.315 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.vWxpnTbEiQ 2024-06-14T23:51:07.453 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.vWxpnTbEiQ is 6KB 2024-06-14T23:51:07.472 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.vWxpnTbEiQ 2024-06-14T23:51:07.485 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-06-14T23:51:07.555 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-14T23:51:07.556 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-14T23:51:07.556 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-14T23:51:07.556 INFO:teuthology.orchestra.run.smithi136.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-14T23:51:07.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-14T23:51:07.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-06-14T23:51:28.521 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-14T23:51:28.522 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:51:28.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:51:37.742 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-14T23:51:37.742 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:51:37.743 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:51:46.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-06-14T23:51:55.982 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:51:55.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:51:56.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-06-14T23:51:59.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-06-14T23:52:05.194 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:52:05.195 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:52:08.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-06-14T23:52:11.042 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:52:11.042 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:52:20.264 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:52:20.266 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:52:29.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2024-06-14T23:52:38.686 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-06-14T23:52:44.474 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:52:44.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:52:45.137 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-06-14T23:52:45.584 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:52:45.584 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-457.el9.x86_64"... 2024-06-14T23:52:45.585 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2024-06-14T23:52:45.600 INFO:teuthology.orchestra.run.smithi063.stdout:5.14.0-457.el9.x86_64 2024-06-14T23:52:45.601 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-14T23:52:45.601 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-14T23:52:45.601 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-14T23:52:46.601 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-14T23:52:46.602 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-14T23:52:46.689 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2024-06-14T23:52:46.747 DEBUG:teuthology.parallel:result is None 2024-06-14T23:52:53.687 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:52:53.688 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:52:54.026 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-06-14T23:52:54.522 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:52:54.522 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-457.el9.x86_64"... 2024-06-14T23:52:54.522 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-06-14T23:52:54.536 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-457.el9.x86_64 2024-06-14T23:52:54.536 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-14T23:52:54.537 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-14T23:52:54.537 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-14T23:52:55.538 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-14T23:52:55.538 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-14T23:52:55.725 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-06-14T23:52:55.757 DEBUG:teuthology.parallel:result is None 2024-06-14T23:52:55.757 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-14T23:52:55.764 INFO:teuthology.task.internal:Creating test directory... 2024-06-14T23:52:55.764 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-14T23:52:55.767 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-14T23:52:55.788 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-14T23:52:55.802 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-14T23:52:55.872 INFO:teuthology.task.internal:Creating archive directory... 2024-06-14T23:52:55.873 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-14T23:52:55.875 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-14T23:52:55.939 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-14T23:52:55.945 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-14T23:52:55.946 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-14T23:52:55.948 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-14T23:52:55.978 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T23:52:55.990 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T23:52:56.127 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T23:52:56.146 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-14T23:52:56.149 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-14T23:52:56.157 INFO:teuthology.task.internal:Configuring sudo... 2024-06-14T23:52:56.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-14T23:52:56.160 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-14T23:52:56.215 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-14T23:52:56.224 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-14T23:52:56.224 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-14T23:52:56.230 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-14T23:52:56.256 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T23:52:56.333 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T23:52:56.384 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T23:52:56.469 DEBUG:teuthology.orchestra.run.smithi063:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T23:52:56.511 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:52:56.511 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-14T23:52:56.578 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T23:52:56.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-14T23:52:56.719 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T23:52:56.784 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-14T23:52:56.827 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:52:56.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-14T23:52:56.895 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2024-06-14T23:52:56.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-06-14T23:52:56.950 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T23:52:56.983 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-14T23:52:57.462 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-14T23:52:57.472 INFO:teuthology.task.internal:Starting timer... 2024-06-14T23:52:57.472 INFO:teuthology.run_tasks:Running task pcp... 2024-06-14T23:52:57.524 INFO:teuthology.run_tasks:Running task selinux... 2024-06-14T23:52:57.563 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-06-14T23:52:57.564 DEBUG:teuthology.orchestra.run.smithi063:> sudo service auditd rotate 2024-06-14T23:52:57.649 INFO:teuthology.orchestra.run.smithi063.stdout:Rotating logs: 2024-06-14T23:52:57.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-06-14T23:52:57.714 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-06-14T23:52:57.715 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-14T23:52:57.716 DEBUG:teuthology.orchestra.run.smithi063:> /usr/sbin/getenforce 2024-06-14T23:52:57.739 INFO:teuthology.orchestra.run.smithi063.stdout:Permissive 2024-06-14T23:52:57.739 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-06-14T23:52:57.763 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-06-14T23:52:57.763 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi063.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-06-14T23:52:57.763 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T23:52:57.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:52:57.809 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-14T23:52:57.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-14T23:52:57.834 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-14T23:52:57.835 DEBUG:teuthology.orchestra.run.smithi063:> sudo /usr/sbin/setenforce permissive 2024-06-14T23:52:57.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-06-14T23:52:57.916 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-14T23:52:57.925 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-14T23:52:57.936 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-14T23:52:57.937 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi063.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-14T23:57:21.747 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi063.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-06-14T23:57:21.748 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:57:21.749 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:57:21.830 DEBUG:teuthology.orchestra.run.smithi063:> true 2024-06-14T23:57:21.923 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi063.front.sepia.ceph.com' 2024-06-14T23:57:21.924 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:57:21.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-14T23:57:21.996 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-06-14T23:57:22.082 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-06-14T23:57:22.083 INFO:teuthology.run_tasks:Running task clock... 2024-06-14T23:57:22.092 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-14T23:57:22.093 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-14T23:57:22.093 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T23:57:22.096 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-14T23:57:22.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-14T23:57:22.126 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-14T23:57:22.143 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-14T23:57:22.152 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-14T23:57:22.167 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-14T23:57:22.169 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: ntpd: command not found 2024-06-14T23:57:22.181 INFO:teuthology.orchestra.run.smithi063.stdout:506 Cannot talk to daemon 2024-06-14T23:57:22.191 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-06-14T23:57:22.196 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-14T23:57:22.203 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-06-14T23:57:22.211 INFO:teuthology.orchestra.run.smithi063.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-14T23:57:22.217 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-14T23:57:22.231 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-14T23:57:22.277 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-06-14T23:57:22.281 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T23:57:22.281 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-06-14T23:57:22.282 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.282 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.282 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.282 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.286 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-06-14T23:57:22.289 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-14T23:57:22.289 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-06-14T23:57:22.289 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.290 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.290 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.290 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-14T23:57:22.291 INFO:teuthology.run_tasks:Running task install... 2024-06-14T23:57:22.298 DEBUG:teuthology.task.install:project ceph 2024-06-14T23:57:22.299 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}} 2024-06-14T23:57:22.299 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-14T23:57:22.299 INFO:teuthology.task.install:Using flavor: default 2024-06-14T23:57:22.304 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-14T23:57:22.304 INFO:teuthology.task.install:extra packages: [] 2024-06-14T23:57:22.305 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-14T23:57:22.305 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:57:22.305 INFO:teuthology.packaging:ref: None 2024-06-14T23:57:22.305 INFO:teuthology.packaging:tag: None 2024-06-14T23:57:22.305 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:57:22.305 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:57:22.305 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T23:57:22.307 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-14T23:57:22.307 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:57:22.307 INFO:teuthology.packaging:ref: None 2024-06-14T23:57:22.307 INFO:teuthology.packaging:tag: None 2024-06-14T23:57:22.307 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:57:22.308 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:57:22.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T23:57:22.549 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-14T23:57:22.549 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-14T23:57:22.601 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-14T23:57:22.602 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-14T23:57:22.667 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-14T23:57:22.667 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:57:22.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-14T23:57:22.693 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-14T23:57:22.693 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:57:22.694 INFO:teuthology.packaging:ref: None 2024-06-14T23:57:22.694 INFO:teuthology.packaging:tag: None 2024-06-14T23:57:22.694 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:57:22.694 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:57:22.694 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-14T23:57:22.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-14T23:57:22.775 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-14T23:57:22.775 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:57:22.775 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-14T23:57:22.803 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-14T23:57:22.804 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:57:22.804 INFO:teuthology.packaging:ref: None 2024-06-14T23:57:22.804 INFO:teuthology.packaging:tag: None 2024-06-14T23:57:22.804 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:57:22.804 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:57:22.804 DEBUG:teuthology.orchestra.run.smithi063:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-14T23:57:22.834 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-14T23:57:22.875 DEBUG:teuthology.orchestra.run.smithi063:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-14T23:57:22.899 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-06-14T23:57:22.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-06-14T23:57:22.951 DEBUG:teuthology.orchestra.run.smithi063:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-14T23:57:23.018 INFO:teuthology.orchestra.run.smithi063.stdout:check_obsoletes = 1 2024-06-14T23:57:23.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum clean all 2024-06-14T23:57:23.130 INFO:teuthology.orchestra.run.smithi136.stdout:45 files removed 2024-06-14T23:57:23.156 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-06-14T23:57:23.260 INFO:teuthology.orchestra.run.smithi063.stdout:45 files removed 2024-06-14T23:57:23.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-radosgw 2024-06-14T23:57:23.848 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 187 kB/s | 77 kB 00:00 2024-06-14T23:57:24.071 INFO:teuthology.orchestra.run.smithi063.stdout:ceph packages for x86_64 158 kB/s | 77 kB 00:00 2024-06-14T23:57:24.183 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 38 kB/s | 11 kB 00:00 2024-06-14T23:57:24.438 INFO:teuthology.orchestra.run.smithi063.stdout:ceph noarch packages 34 kB/s | 11 kB 00:00 2024-06-14T23:57:24.549 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-06-14T23:57:24.755 INFO:teuthology.orchestra.run.smithi063.stdout:ceph source packages 6.6 kB/s | 1.8 kB 00:00 2024-06-14T23:57:25.216 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-14T23:57:26.415 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-06-14T23:57:29.984 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 5.4 MB/s | 20 MB 00:03 2024-06-14T23:57:30.805 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 20 MB 00:03 2024-06-14T23:57:34.068 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.3 MB 00:00 2024-06-14T23:57:35.693 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 17 kB 00:00 2024-06-14T23:57:36.165 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.3 MB 00:01 2024-06-14T23:57:36.698 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-06-14T23:57:37.724 INFO:teuthology.orchestra.run.smithi063.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 17 kB 00:00 2024-06-14T23:57:38.678 INFO:teuthology.orchestra.run.smithi063.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-06-14T23:57:42.012 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-06-14T23:57:44.117 INFO:teuthology.orchestra.run.smithi063.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-14T23:57:44.511 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:57:44.524 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:57:44.524 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:57:44.524 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T23:57:44.525 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-14T23:57:44.526 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-14T23:57:44.527 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-14T23:57:44.528 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout:Install 22 Packages 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-06-14T23:57:44.529 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:57:44.530 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 58 M 2024-06-14T23:57:44.530 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:57:45.946 INFO:teuthology.orchestra.run.smithi136.stdout:(1/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 4.2 MB/s | 5.0 MB 00:01 2024-06-14T23:57:46.064 INFO:teuthology.orchestra.run.smithi136.stdout:(2/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 216 kB/s | 25 kB 00:00 2024-06-14T23:57:46.596 INFO:teuthology.orchestra.run.smithi136.stdout:(3/24): ceph-common-17.2.7-1008.ge47ff4b8.el9.x 12 MB/s | 22 MB 00:01 2024-06-14T23:57:46.621 INFO:teuthology.orchestra.run.smithi136.stdout:(4/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 1.1 MB/s | 638 kB 00:00 2024-06-14T23:57:46.652 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrading: 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:57:46.664 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-14T23:57:46.665 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-14T23:57:46.666 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout:Install 22 Packages 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout:Upgrade 2 Packages 2024-06-14T23:57:46.667 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:57:46.669 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 58 M 2024-06-14T23:57:46.669 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:57:46.780 INFO:teuthology.orchestra.run.smithi136.stdout:(5/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9. 6.9 MB/s | 14 MB 00:02 2024-06-14T23:57:46.805 INFO:teuthology.orchestra.run.smithi136.stdout:(6/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 2.1 MB/s | 457 kB 00:00 2024-06-14T23:57:46.839 INFO:teuthology.orchestra.run.smithi136.stdout:(7/24): python3-ceph-argparse-17.2.7-1008.ge47f 774 kB/s | 45 kB 00:00 2024-06-14T23:57:46.898 INFO:teuthology.orchestra.run.smithi136.stdout:(8/24): python3-ceph-common-17.2.7-1008.ge47ff4 1.2 MB/s | 118 kB 00:00 2024-06-14T23:57:46.940 INFO:teuthology.orchestra.run.smithi136.stdout:(9/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 1.5 MB/s | 156 kB 00:00 2024-06-14T23:57:46.965 INFO:teuthology.orchestra.run.smithi136.stdout:(10/24): python3-rados-17.2.7-1008.ge47ff4b8.el 4.7 MB/s | 320 kB 00:00 2024-06-14T23:57:47.024 INFO:teuthology.orchestra.run.smithi136.stdout:(11/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 3.4 MB/s | 292 kB 00:00 2024-06-14T23:57:47.049 INFO:teuthology.orchestra.run.smithi136.stdout:(12/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 1.2 MB/s | 100 kB 00:00 2024-06-14T23:57:47.141 INFO:teuthology.orchestra.run.smithi136.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-06-14T23:57:47.175 INFO:teuthology.orchestra.run.smithi136.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-06-14T23:57:47.233 INFO:teuthology.orchestra.run.smithi136.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-06-14T23:57:47.267 INFO:teuthology.orchestra.run.smithi136.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-06-14T23:57:47.284 INFO:teuthology.orchestra.run.smithi136.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 839 kB/s | 42 kB 00:00 2024-06-14T23:57:47.368 INFO:teuthology.orchestra.run.smithi136.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-06-14T23:57:47.393 INFO:teuthology.orchestra.run.smithi136.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-06-14T23:57:47.427 INFO:teuthology.orchestra.run.smithi136.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-06-14T23:57:47.477 INFO:teuthology.orchestra.run.smithi136.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-06-14T23:57:47.643 INFO:teuthology.orchestra.run.smithi063.stdout:(1/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 7.2 MB/s | 5.0 MB 00:00 2024-06-14T23:57:47.735 INFO:teuthology.orchestra.run.smithi063.stdout:(2/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 275 kB/s | 25 kB 00:00 2024-06-14T23:57:47.777 INFO:teuthology.orchestra.run.smithi136.stdout:(22/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 8.8 MB/s | 3.1 MB 00:00 2024-06-14T23:57:47.844 INFO:teuthology.orchestra.run.smithi063.stdout:(3/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 5.8 MB/s | 638 kB 00:00 2024-06-14T23:57:47.919 INFO:teuthology.orchestra.run.smithi063.stdout:(4/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 6.0 MB/s | 457 kB 00:00 2024-06-14T23:57:48.411 INFO:teuthology.orchestra.run.smithi063.stdout:(5/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_6 10 MB/s | 4.9 MB 00:00 2024-06-14T23:57:48.461 INFO:teuthology.orchestra.run.smithi063.stdout:(6/24): python3-ceph-argparse-17.2.7-1008.ge47f 904 kB/s | 45 kB 00:00 2024-06-14T23:57:48.469 INFO:teuthology.orchestra.run.smithi136.stdout:(23/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_ 2.7 MB/s | 4.9 MB 00:01 2024-06-14T23:57:48.519 INFO:teuthology.orchestra.run.smithi063.stdout:(7/24): python3-ceph-common-17.2.7-1008.ge47ff4 2.0 MB/s | 118 kB 00:00 2024-06-14T23:57:48.578 INFO:teuthology.orchestra.run.smithi063.stdout:(8/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 2.6 MB/s | 156 kB 00:00 2024-06-14T23:57:48.652 INFO:teuthology.orchestra.run.smithi136.stdout:(24/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 2.5 MB/s | 2.9 MB 00:01 2024-06-14T23:57:48.658 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:57:48.658 INFO:teuthology.orchestra.run.smithi136.stdout:Total 14 MB/s | 58 MB 00:04 2024-06-14T23:57:48.723 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:57:48.728 INFO:teuthology.orchestra.run.smithi063.stdout:(9/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9. 7.9 MB/s | 14 MB 00:01 2024-06-14T23:57:48.753 INFO:teuthology.orchestra.run.smithi063.stdout:(10/24): python3-rados-17.2.7-1008.ge47ff4b8.el 1.8 MB/s | 320 kB 00:00 2024-06-14T23:57:48.757 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:57:48.757 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:57:48.779 INFO:teuthology.orchestra.run.smithi063.stdout:(11/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 5.7 MB/s | 292 kB 00:00 2024-06-14T23:57:48.804 INFO:teuthology.orchestra.run.smithi063.stdout:(12/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 1.9 MB/s | 100 kB 00:00 2024-06-14T23:57:49.284 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:57:49.284 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:57:49.430 INFO:teuthology.orchestra.run.smithi063.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-06-14T23:57:49.530 INFO:teuthology.orchestra.run.smithi063.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 213 kB/s | 160 kB 00:00 2024-06-14T23:57:49.789 INFO:teuthology.orchestra.run.smithi063.stdout:(15/24): ceph-common-17.2.7-1008.ge47ff4b8.el9. 7.9 MB/s | 22 MB 00:02 2024-06-14T23:57:50.447 INFO:teuthology.orchestra.run.smithi063.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 64 kB/s | 42 kB 00:00 2024-06-14T23:57:50.469 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:57:50.514 INFO:teuthology.orchestra.run.smithi063.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.6 MB/s | 309 kB 00:00 2024-06-14T23:57:50.539 INFO:teuthology.orchestra.run.smithi063.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 290 kB/s | 292 kB 00:01 2024-06-14T23:57:50.564 INFO:teuthology.orchestra.run.smithi063.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-06-14T23:57:50.623 INFO:teuthology.orchestra.run.smithi063.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 555 kB/s | 662 kB 00:01 2024-06-14T23:57:50.648 INFO:teuthology.orchestra.run.smithi063.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 619 kB/s | 67 kB 00:00 2024-06-14T23:57:50.699 INFO:teuthology.orchestra.run.smithi063.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-06-14T23:57:50.848 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-14T23:57:50.913 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:57:50.940 INFO:teuthology.orchestra.run.smithi063.stdout:(23/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 9.8 MB/s | 3.1 MB 00:00 2024-06-14T23:57:50.991 INFO:teuthology.orchestra.run.smithi063.stdout:(24/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 8.6 MB/s | 2.9 MB 00:00 2024-06-14T23:57:50.995 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:57:50.995 INFO:teuthology.orchestra.run.smithi063.stdout:Total 13 MB/s | 58 MB 00:04 2024-06-14T23:57:51.006 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:57:51.056 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:57:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T23:57:51.094 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:57:51.094 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:57:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:57:51.199 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:57:51.241 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-14T23:57:51.270 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-14T23:57:51.639 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:57:51.639 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:57:51.702 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-14T23:57:51.726 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T23:57:51.789 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T23:57:51.817 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-14T23:57:51.852 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-14T23:57:51.898 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T23:57:51.960 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T23:57:51.987 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T23:57:52.052 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T23:57:52.103 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-14T23:57:52.138 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-14T23:57:52.170 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-14T23:57:52.464 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-14T23:57:52.508 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T23:57:52.582 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T23:57:52.655 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-14T23:57:52.686 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-14T23:57:52.826 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:57:53.226 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-14T23:57:53.276 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:57:53.379 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:57:53.475 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T23:57:53.502 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:57:53.591 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:57:53.638 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-14T23:57:53.674 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-14T23:57:54.132 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-14T23:57:54.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T23:57:54.227 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-14T23:57:54.264 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-14T23:57:54.307 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-14T23:57:54.345 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T23:57:54.408 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T23:57:54.433 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T23:57:54.491 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-14T23:57:54.531 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-14T23:57:54.567 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-14T23:57:54.599 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-14T23:57:54.908 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-14T23:57:54.946 INFO:teuthology.orchestra.run.smithi063.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T23:57:55.020 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-14T23:57:55.097 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-14T23:57:55.124 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-14T23:57:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:55.711 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:55.748 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T23:57:55.823 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T23:57:55.823 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-14T23:57:55.823 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-14T23:57:55.823 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:57:55.850 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:57:57.854 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:57.895 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:58.448 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-14T23:57:58.490 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T23:57:58.593 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-14T23:57:58.593 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-14T23:57:58.593 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-14T23:57:58.593 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:57:58.624 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:58:05.367 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-06-14T23:58:05.368 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:06.679 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T23:58:06.757 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T23:58:06.757 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:06.758 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-14T23:58:06.758 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T23:58:06.758 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T23:58:06.758 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:06.759 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T23:58:06.840 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T23:58:06.840 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-14T23:58:08.132 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-14T23:58:08.133 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-14T23:58:08.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-14T23:58:08.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:58:08.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-14T23:58:08.134 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-14T23:58:08.534 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:58:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /sys 2024-06-14T23:58:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /proc 2024-06-14T23:58:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /mnt 2024-06-14T23:58:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /var/tmp 2024-06-14T23:58:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /home 2024-06-14T23:58:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /root 2024-06-14T23:58:08.535 INFO:teuthology.orchestra.run.smithi063.stdout:skipping the directory /tmp 2024-06-14T23:58:08.535 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:08.845 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-14T23:58:08.846 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-14T23:58:08.847 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.848 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.848 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:08.848 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-14T23:58:08.848 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:08.848 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:09.180 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-06-14T23:58:09.767 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:28 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:09.841 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T23:58:09.891 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-14T23:58:09.892 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 192 M 2024-06-14T23:58:09.893 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:09.920 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-14T23:58:09.920 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:09.920 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-14T23:58:09.920 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T23:58:09.921 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-14T23:58:09.921 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:09.922 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T23:58:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-14T23:58:10.010 INFO:teuthology.orchestra.run.smithi063.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:10.170 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-06-14T23:58:10.211 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-14T23:58:10.320 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 533 kB/s | 163 kB 00:00 2024-06-14T23:58:11.364 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:11.364 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-14T23:58:11.364 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-14T23:58:11.365 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-14T23:58:11.366 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-14T23:58:12.086 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout:Upgraded: 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-14T23:58:12.087 INFO:teuthology.orchestra.run.smithi063.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:12.088 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:12.362 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 20 MB/s | 48 MB 00:02 2024-06-14T23:58:12.364 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:12.364 INFO:teuthology.orchestra.run.smithi136.stdout:Total 20 MB/s | 48 MB 00:02 2024-06-14T23:58:12.392 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:12.412 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:12.412 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:12.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-test 2024-06-14T23:58:12.777 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:12.777 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:13.045 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:29 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:13.175 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:13.175 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:13.175 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-14T23:58:13.176 INFO:teuthology.orchestra.run.smithi063.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout:Install 4 Packages 2024-06-14T23:58:13.177 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:13.178 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 48 M 2024-06-14T23:58:13.178 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 192 M 2024-06-14T23:58:13.179 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:13.294 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:13.386 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-14T23:58:13.443 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-14T23:58:13.466 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T23:58:13.480 INFO:teuthology.orchestra.run.smithi063.stdout:(1/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 693 kB/s | 163 kB 00:00 2024-06-14T23:58:13.822 INFO:teuthology.orchestra.run.smithi063.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-06-14T23:58:13.848 INFO:teuthology.orchestra.run.smithi063.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 505 kB/s | 304 kB 00:00 2024-06-14T23:58:15.740 INFO:teuthology.orchestra.run.smithi063.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 19 MB/s | 48 MB 00:02 2024-06-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:15.742 INFO:teuthology.orchestra.run.smithi063.stdout:Total 19 MB/s | 48 MB 00:02 2024-06-14T23:58:15.770 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:15.791 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:15.791 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:16.176 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:16.176 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:16.715 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:16.796 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-14T23:58:16.847 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-14T23:58:16.866 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T23:58:17.864 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T23:58:17.958 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T23:58:20.205 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T23:58:20.205 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-14T23:58:20.205 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-14T23:58:20.206 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:20.477 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:20.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-06-14T23:58:21.364 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:40 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:21.495 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:21.497 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:21.497 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:21.497 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:21.497 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-14T23:58:21.498 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-14T23:58:21.499 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout:Install 33 Packages 2024-06-14T23:58:21.500 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:21.501 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 27 M 2024-06-14T23:58:21.501 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 96 M 2024-06-14T23:58:21.501 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:21.530 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-14T23:58:21.608 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T23:58:22.181 INFO:teuthology.orchestra.run.smithi136.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-06-14T23:58:22.531 INFO:teuthology.orchestra.run.smithi136.stdout:(2/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-06-14T23:58:22.689 INFO:teuthology.orchestra.run.smithi136.stdout:(3/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 3.1 MB/s | 2.1 MB 00:00 2024-06-14T23:58:22.723 INFO:teuthology.orchestra.run.smithi136.stdout:(4/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 2.0 MB/s | 1.4 MB 00:00 2024-06-14T23:58:22.790 INFO:teuthology.orchestra.run.smithi136.stdout:(5/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 2.3 MB/s | 240 kB 00:00 2024-06-14T23:58:22.865 INFO:teuthology.orchestra.run.smithi136.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-06-14T23:58:22.907 INFO:teuthology.orchestra.run.smithi136.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-06-14T23:58:22.949 INFO:teuthology.orchestra.run.smithi136.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 3.2 MB/s | 135 kB 00:00 2024-06-14T23:58:22.991 INFO:teuthology.orchestra.run.smithi136.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-06-14T23:58:23.033 INFO:teuthology.orchestra.run.smithi136.stdout:(10/33): python3-urllib3-1.26.5-5.el9.noarch.rp 5.1 MB/s | 215 kB 00:00 2024-06-14T23:58:23.108 INFO:teuthology.orchestra.run.smithi136.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 3.9 MB/s | 1.2 MB 00:00 2024-06-14T23:58:23.167 INFO:teuthology.orchestra.run.smithi136.stdout:(12/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-06-14T23:58:23.192 INFO:teuthology.orchestra.run.smithi136.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-06-14T23:58:23.217 INFO:teuthology.orchestra.run.smithi136.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-14T23:58:23.267 INFO:teuthology.orchestra.run.smithi136.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 554 kB/s | 42 kB 00:00 2024-06-14T23:58:23.443 INFO:teuthology.orchestra.run.smithi136.stdout:(16/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86 17 MB/s | 15 MB 00:00 2024-06-14T23:58:23.468 INFO:teuthology.orchestra.run.smithi136.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 173 kB/s | 43 kB 00:00 2024-06-14T23:58:23.518 INFO:teuthology.orchestra.run.smithi136.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 430 kB/s | 11 kB 00:00 2024-06-14T23:58:23.543 INFO:teuthology.orchestra.run.smithi136.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 628 kB/s | 172 kB 00:00 2024-06-14T23:58:23.569 INFO:teuthology.orchestra.run.smithi136.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 369 kB/s | 18 kB 00:00 2024-06-14T23:58:23.594 INFO:teuthology.orchestra.run.smithi136.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-06-14T23:58:23.619 INFO:teuthology.orchestra.run.smithi136.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-06-14T23:58:23.644 INFO:teuthology.orchestra.run.smithi136.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-06-14T23:58:23.670 INFO:teuthology.orchestra.run.smithi136.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-06-14T23:58:23.695 INFO:teuthology.orchestra.run.smithi136.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-14T23:58:23.728 INFO:teuthology.orchestra.run.smithi136.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-06-14T23:58:23.739 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-14T23:58:23.739 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-14T23:58:23.739 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-14T23:58:23.739 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-14T23:58:23.754 INFO:teuthology.orchestra.run.smithi136.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-06-14T23:58:23.779 INFO:teuthology.orchestra.run.smithi136.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-14T23:58:23.808 INFO:teuthology.orchestra.run.smithi136.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 670 kB/s | 36 kB 00:00 2024-06-14T23:58:23.838 INFO:teuthology.orchestra.run.smithi136.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-06-14T23:58:23.855 INFO:teuthology.orchestra.run.smithi136.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-06-14T23:58:23.880 INFO:teuthology.orchestra.run.smithi136.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-06-14T23:58:23.905 INFO:teuthology.orchestra.run.smithi136.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.3 MB/s | 427 kB 00:00 2024-06-14T23:58:23.909 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:23.910 INFO:teuthology.orchestra.run.smithi136.stdout:Total 11 MB/s | 27 MB 00:02 2024-06-14T23:58:24.031 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:24.032 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:24.035 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:24.094 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:24.094 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:24.256 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph 2024-06-14T23:58:24.470 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:24.471 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:24.847 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:40 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:24.982 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:24.984 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:24.984 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:58:24.984 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:24.984 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-14T23:58:24.985 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-14T23:58:24.986 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout:Install 33 Packages 2024-06-14T23:58:24.987 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:24.988 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 27 M 2024-06-14T23:58:24.988 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 96 M 2024-06-14T23:58:24.988 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:25.515 INFO:teuthology.orchestra.run.smithi063.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 28 kB/s | 6.5 kB 00:00 2024-06-14T23:58:25.645 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:25.673 INFO:teuthology.orchestra.run.smithi063.stdout:(2/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 5.3 MB/s | 2.1 MB 00:00 2024-06-14T23:58:25.700 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-14T23:58:25.715 INFO:teuthology.orchestra.run.smithi063.stdout:(3/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 3.3 MB/s | 1.4 MB 00:00 2024-06-14T23:58:25.732 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-14T23:58:25.773 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-14T23:58:25.799 INFO:teuthology.orchestra.run.smithi063.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 2.8 MB/s | 240 kB 00:00 2024-06-14T23:58:25.865 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-14T23:58:25.952 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-14T23:58:25.974 INFO:teuthology.orchestra.run.smithi063.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-06-14T23:58:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-14T23:58:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-14T23:58:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-14T23:58:26.091 INFO:teuthology.orchestra.run.smithi063.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-06-14T23:58:26.108 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-14T23:58:26.141 INFO:teuthology.orchestra.run.smithi063.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-14T23:58:26.142 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-14T23:58:26.182 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-14T23:58:26.191 INFO:teuthology.orchestra.run.smithi063.stdout:(8/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-06-14T23:58:26.226 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-14T23:58:26.242 INFO:teuthology.orchestra.run.smithi063.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.4 MB/s | 125 kB 00:00 2024-06-14T23:58:26.251 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-14T23:58:26.290 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-14T23:58:26.364 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-14T23:58:26.401 INFO:teuthology.orchestra.run.smithi063.stdout:(10/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86 21 MB/s | 15 MB 00:00 2024-06-14T23:58:26.426 INFO:teuthology.orchestra.run.smithi063.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-06-14T23:58:26.509 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-14T23:58:26.559 INFO:teuthology.orchestra.run.smithi063.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 261 kB/s | 35 kB 00:00 2024-06-14T23:58:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-14T23:58:26.601 INFO:teuthology.orchestra.run.smithi063.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 860 kB/s | 172 kB 00:00 2024-06-14T23:58:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-14T23:58:26.651 INFO:teuthology.orchestra.run.smithi063.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 555 kB/s | 51 kB 00:00 2024-06-14T23:58:26.679 INFO:teuthology.orchestra.run.smithi063.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 538 kB/s | 42 kB 00:00 2024-06-14T23:58:26.681 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-14T23:58:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-14T23:58:26.763 INFO:teuthology.orchestra.run.smithi063.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 391 kB/s | 43 kB 00:00 2024-06-14T23:58:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-14T23:58:26.796 INFO:teuthology.orchestra.run.smithi063.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-06-14T23:58:26.821 INFO:teuthology.orchestra.run.smithi063.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-14T23:58:26.847 INFO:teuthology.orchestra.run.smithi063.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-06-14T23:58:26.872 INFO:teuthology.orchestra.run.smithi063.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-06-14T23:58:26.897 INFO:teuthology.orchestra.run.smithi063.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 460 kB/s | 23 kB 00:00 2024-06-14T23:58:26.922 INFO:teuthology.orchestra.run.smithi063.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-06-14T23:58:26.948 INFO:teuthology.orchestra.run.smithi063.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-06-14T23:58:26.965 INFO:teuthology.orchestra.run.smithi063.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-06-14T23:58:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-14T23:58:26.990 INFO:teuthology.orchestra.run.smithi063.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-06-14T23:58:27.015 INFO:teuthology.orchestra.run.smithi063.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 656 kB/s | 16 kB 00:00 2024-06-14T23:58:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-14T23:58:27.040 INFO:teuthology.orchestra.run.smithi063.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-06-14T23:58:27.066 INFO:teuthology.orchestra.run.smithi063.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-14T23:58:27.082 INFO:teuthology.orchestra.run.smithi063.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 2.2 MB/s | 36 kB 00:00 2024-06-14T23:58:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-14T23:58:27.116 INFO:teuthology.orchestra.run.smithi063.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-06-14T23:58:27.141 INFO:teuthology.orchestra.run.smithi063.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-06-14T23:58:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-14T23:58:27.208 INFO:teuthology.orchestra.run.smithi063.stdout:(32/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86 2.3 MB/s | 3.9 MB 00:01 2024-06-14T23:58:27.242 INFO:teuthology.orchestra.run.smithi063.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-06-14T23:58:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-14T23:58:27.246 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:27.246 INFO:teuthology.orchestra.run.smithi063.stdout:Total 12 MB/s | 27 MB 00:02 2024-06-14T23:58:27.362 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-14T23:58:27.402 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:27.403 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:27.512 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-14T23:58:27.535 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T23:58:27.561 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T23:58:27.561 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:27.561 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-14T23:58:27.561 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T23:58:27.562 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T23:58:27.562 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:27.789 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:27.789 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T23:58:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:28.946 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:29.003 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-14T23:58:29.036 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-14T23:58:29.069 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-14T23:58:29.164 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-14T23:58:29.250 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-14T23:58:29.295 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T23:58:29.306 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T23:58:29.319 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:29.349 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-14T23:58:29.378 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-14T23:58:29.413 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-14T23:58:29.446 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-14T23:58:29.487 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-14T23:58:29.529 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T23:58:29.530 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T23:58:29.553 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:29.564 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-14T23:58:29.602 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-14T23:58:29.618 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T23:58:29.680 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-14T23:58:29.838 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-14T23:58:29.931 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-14T23:58:29.981 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-14T23:58:30.020 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-14T23:58:30.078 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-14T23:58:30.142 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-14T23:58:30.346 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-14T23:58:30.398 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-14T23:58:30.479 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-14T23:58:30.569 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-14T23:58:30.614 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-14T23:58:30.737 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-14T23:58:30.891 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-14T23:58:30.916 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-14T23:58:30.943 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:31.253 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-14T23:58:31.254 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-14T23:58:31.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-14T23:58:31.256 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-14T23:58:31.257 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-14T23:58:31.257 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-14T23:58:31.257 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-14T23:58:31.257 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-14T23:58:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-14T23:58:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-14T23:58:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-14T23:58:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:32.258 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:32.353 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T23:58:32.384 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-14T23:58:32.384 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:32.384 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-14T23:58:32.385 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T23:58:32.385 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-14T23:58:32.385 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:32.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-06-14T23:58:32.771 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-14T23:58:32.797 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:33.014 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T23:58:33.038 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-14T23:58:33.039 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:33.039 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-14T23:58:33.039 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T23:58:33.039 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-14T23:58:33.039 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:33.105 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T23:58:33.186 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:52 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:33.270 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:58:33.314 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:33.315 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:58:33.315 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:33.388 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-06-14T23:58:33.969 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:52 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:34.101 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:34.102 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:58:34.103 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:34.103 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 82 k 2024-06-14T23:58:34.103 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 372 k 2024-06-14T23:58:34.103 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:34.320 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 377 kB/s | 82 kB 00:00 2024-06-14T23:58:34.321 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:34.321 INFO:teuthology.orchestra.run.smithi136.stdout:Total 374 kB/s | 82 kB 00:00 2024-06-14T23:58:34.321 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:34.326 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:34.326 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:34.328 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:34.328 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:34.397 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:34.671 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-14T23:58:34.672 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-14T23:58:34.673 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-14T23:58:34.674 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-14T23:58:34.844 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:34.956 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:35.543 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-14T23:58:35.543 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:35.543 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:35.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:35.543 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-14T23:58:35.544 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ply-3.11-14.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-14T23:58:35.545 INFO:teuthology.orchestra.run.smithi063.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-14T23:58:35.546 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-14T23:58:35.546 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-14T23:58:35.546 INFO:teuthology.orchestra.run.smithi063.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-14T23:58:35.546 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:35.546 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:35.560 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:35.791 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-06-14T23:58:35.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-base 2024-06-14T23:58:36.370 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:55 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:36.502 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 137 k 2024-06-14T23:58:36.503 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 379 k 2024-06-14T23:58:36.504 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:36.509 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:52 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:36.596 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:58:36.641 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:36.642 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:58:36.642 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:36.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install cephadm 2024-06-14T23:58:36.834 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 416 kB/s | 137 kB 00:00 2024-06-14T23:58:36.834 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:36.834 INFO:teuthology.orchestra.run.smithi136.stdout:Total 414 kB/s | 137 kB 00:00 2024-06-14T23:58:36.835 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:36.842 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:36.842 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:36.899 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:36.899 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:37.054 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:37.135 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:37.135 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-14T23:58:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:37.304 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:53 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:37.435 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:37.436 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:58:37.437 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 82 k 2024-06-14T23:58:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 372 k 2024-06-14T23:58:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:37.641 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 406 kB/s | 82 kB 00:00 2024-06-14T23:58:37.642 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:37.642 INFO:teuthology.orchestra.run.smithi063.stdout:Total 402 kB/s | 82 kB 00:00 2024-06-14T23:58:37.642 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:37.646 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:37.648 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:37.648 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:37.723 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:38.059 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:38.059 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:38.060 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:38.060 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:38.060 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:38.060 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:38.174 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:38.279 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-06-14T23:58:38.287 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:38.598 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:38.850 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:38.867 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:57 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:38.954 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:58:38.996 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:38.997 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:58:38.997 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:39.062 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-immutable-object-cache 2024-06-14T23:58:39.070 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-06-14T23:58:39.643 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:00:58 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:39.661 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:55 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:39.771 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:39.771 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:39.771 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-14T23:58:39.772 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:Install 7 Packages 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.7 M 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 80 M 2024-06-14T23:58:39.773 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:39.793 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:39.793 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 137 k 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 379 k 2024-06-14T23:58:39.794 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:40.114 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 431 kB/s | 137 kB 00:00 2024-06-14T23:58:40.114 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:40.114 INFO:teuthology.orchestra.run.smithi063.stdout:Total 429 kB/s | 137 kB 00:00 2024-06-14T23:58:40.114 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:40.122 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:40.122 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:40.181 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:40.182 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:40.341 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:40.414 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:40.437 INFO:teuthology.orchestra.run.smithi136.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1008.ge47 99 kB/s | 24 kB 00:00 2024-06-14T23:58:40.440 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:40.440 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:58:40.440 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-14T23:58:40.440 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:40.487 INFO:teuthology.orchestra.run.smithi136.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 49 kB/s | 15 kB 00:00 2024-06-14T23:58:40.512 INFO:teuthology.orchestra.run.smithi136.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 121 kB/s | 9.0 kB 00:00 2024-06-14T23:58:40.562 INFO:teuthology.orchestra.run.smithi136.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.2 MB/s | 31 kB 00:00 2024-06-14T23:58:40.588 INFO:teuthology.orchestra.run.smithi136.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 416 kB/s | 41 kB 00:00 2024-06-14T23:58:40.613 INFO:teuthology.orchestra.run.smithi136.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.8 MB/s | 188 kB 00:00 2024-06-14T23:58:41.188 INFO:teuthology.orchestra.run.smithi136.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 3.4 MB/s | 3.4 MB 00:00 2024-06-14T23:58:41.192 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:41.192 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.6 MB/s | 3.7 MB 00:01 2024-06-14T23:58:41.192 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:41.200 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:41.200 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:41.264 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:41.264 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:41.356 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-14T23:58:41.357 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:41.357 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:41.357 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:58:41.357 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:41.357 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:41.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr 2024-06-14T23:58:41.572 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-14T23:58:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-14T23:58:41.711 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-14T23:58:41.743 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-14T23:58:41.772 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-14T23:58:42.158 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:58 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:42.245 INFO:teuthology.orchestra.run.smithi063.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:58:42.289 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:42.290 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:58:42.290 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:42.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-dashboard 2024-06-14T23:58:42.751 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-14T23:58:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T23:58:42.956 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:00:58 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:43.088 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout:Installing weak dependencies: 2024-06-14T23:58:43.089 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Install 7 Packages 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 3.7 M 2024-06-14T23:58:43.090 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 80 M 2024-06-14T23:58:43.091 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:43.534 INFO:teuthology.orchestra.run.smithi063.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 54 kB/s | 15 kB 00:00 2024-06-14T23:58:43.559 INFO:teuthology.orchestra.run.smithi063.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1008.ge47 82 kB/s | 24 kB 00:00 2024-06-14T23:58:43.643 INFO:teuthology.orchestra.run.smithi063.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-06-14T23:58:43.668 INFO:teuthology.orchestra.run.smithi063.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-14T23:58:43.685 INFO:teuthology.orchestra.run.smithi063.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 326 kB/s | 41 kB 00:00 2024-06-14T23:58:43.710 INFO:teuthology.orchestra.run.smithi063.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 4.4 MB/s | 188 kB 00:00 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-14T23:58:43.900 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-14T23:58:44.219 INFO:teuthology.orchestra.run.smithi063.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 3.5 MB/s | 3.4 MB 00:00 2024-06-14T23:58:44.222 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:44.223 INFO:teuthology.orchestra.run.smithi063.stdout:Total 3.2 MB/s | 3.7 MB 00:01 2024-06-14T23:58:44.223 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:44.232 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:44.232 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-14T23:58:44.233 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:44.234 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:44.298 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:44.299 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:44.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-14T23:58:44.603 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:44.662 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-14T23:58:44.696 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-14T23:58:44.744 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-14T23:58:44.775 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-14T23:58:44.812 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-14T23:58:44.989 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:03 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:45.117 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-14T23:58:45.118 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:45.119 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================= 2024-06-14T23:58:45.120 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-06-14T23:58:45.120 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:45.120 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 38 M 2024-06-14T23:58:45.120 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-06-14T23:58:45.120 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:45.437 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-06-14T23:58:45.512 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-06-14T23:58:45.862 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-14T23:58:45.871 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 75 kB/s | 33 kB 00:00 2024-06-14T23:58:45.907 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T23:58:45.996 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 145 kB/s | 18 kB 00:00 2024-06-14T23:58:46.146 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 235 kB/s | 35 kB 00:00 2024-06-14T23:58:46.463 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.1 MB/s | 3.0 MB 00:00 2024-06-14T23:58:46.830 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): ceph-mgr-diskprediction-local-17.2.7-10 4.8 MB/s | 7.4 MB 00:01 2024-06-14T23:58:46.889 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-14T23:58:46.889 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-14T23:58:46.890 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-14T23:58:46.890 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-14T23:58:46.890 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-14T23:58:46.890 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-14T23:58:46.890 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-14T23:58:46.906 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 6.2 MB/s | 4.7 MB 00:00 2024-06-14T23:58:46.931 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-devel-3.9.18-3.el9.x86_64.rpm 523 kB/s | 244 kB 00:00 2024-06-14T23:58:47.089 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-06-14T23:58:47.114 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.3 MB/s | 250 kB 00:00 2024-06-14T23:58:47.199 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-14T23:58:47.199 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:47.199 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:47.200 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:58:47.399 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-14T23:58:47.598 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 24 MB/s | 16 MB 00:00 2024-06-14T23:58:47.601 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:47.601 INFO:teuthology.orchestra.run.smithi136.stdout:Total 15 MB/s | 38 MB 00:02 2024-06-14T23:58:47.955 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:47.982 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:47.982 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:47.998 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:03 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:58:48.131 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-14T23:58:48.132 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout:======================================================================================= 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout:Install 12 Packages 2024-06-14T23:58:48.133 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:58:48.134 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 38 M 2024-06-14T23:58:48.134 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 209 M 2024-06-14T23:58:48.134 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:58:48.484 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:48.484 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:48.711 INFO:teuthology.orchestra.run.smithi063.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 835 kB/s | 195 kB 00:00 2024-06-14T23:58:48.786 INFO:teuthology.orchestra.run.smithi063.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 803 kB 00:00 2024-06-14T23:58:48.811 INFO:teuthology.orchestra.run.smithi063.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-06-14T23:58:48.853 INFO:teuthology.orchestra.run.smithi063.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-06-14T23:58:48.904 INFO:teuthology.orchestra.run.smithi063.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 702 kB/s | 35 kB 00:00 2024-06-14T23:58:49.146 INFO:teuthology.orchestra.run.smithi063.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 11 MB/s | 7.4 MB 00:00 2024-06-14T23:58:49.244 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:49.279 INFO:teuthology.orchestra.run.smithi063.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-06-14T23:58:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-14T23:58:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-14T23:58:49.461 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-14T23:58:49.505 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-14T23:58:49.538 INFO:teuthology.orchestra.run.smithi063.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.0 MB/s | 3.0 MB 00:00 2024-06-14T23:58:49.630 INFO:teuthology.orchestra.run.smithi063.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.7 MB/s | 250 kB 00:00 2024-06-14T23:58:49.691 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-14T23:58:49.721 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-14T23:58:49.815 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-14T23:58:50.130 INFO:teuthology.orchestra.run.smithi063.stdout:(10/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 3.9 MB/s | 4.7 MB 00:01 2024-06-14T23:58:50.455 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-14T23:58:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-14T23:58:50.697 INFO:teuthology.orchestra.run.smithi063.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.6 MB/s | 5.1 MB 00:01 2024-06-14T23:58:51.802 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T23:58:52.840 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-14T23:58:52.869 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T23:58:54.280 INFO:teuthology.orchestra.run.smithi063.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 3.5 MB/s | 16 MB 00:04 2024-06-14T23:58:54.281 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:54.282 INFO:teuthology.orchestra.run.smithi063.stdout:Total 6.2 MB/s | 38 MB 00:06 2024-06-14T23:58:54.652 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:58:54.678 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:58:54.679 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:58:55.198 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:58:55.199 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:58:55.413 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T23:58:55.413 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T23:58:55.414 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-14T23:58:55.808 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:55.809 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:58:55.986 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:58:56.060 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-14T23:58:56.104 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-14T23:58:56.208 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-14T23:58:56.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-06-14T23:58:56.254 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-14T23:58:56.444 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-14T23:58:56.487 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-14T23:58:56.582 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-14T23:58:56.840 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:15 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:58:56.968 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-14T23:58:56.969 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:56.970 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.2 M 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-06-14T23:58:56.971 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:58:57.272 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-14T23:58:57.396 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-14T23:58:57.407 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 277 kB/s | 26 kB 00:00 2024-06-14T23:58:57.424 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 175 kB/s | 19 kB 00:00 2024-06-14T23:58:57.541 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-06-14T23:58:57.566 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-06-14T23:58:57.596 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 168 kB/s | 47 kB 00:00 2024-06-14T23:58:57.621 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 976 kB/s | 54 kB 00:00 2024-06-14T23:58:57.705 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 279 kB 00:00 2024-06-14T23:58:57.730 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 240 kB/s | 32 kB 00:00 2024-06-14T23:58:57.781 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-06-14T23:58:57.806 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-06-14T23:58:57.831 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-06-14T23:58:57.856 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-google-auth-2.29.0-2.el9.noarc 1.4 MB/s | 205 kB 00:00 2024-06-14T23:58:57.890 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.5 MB/s | 1.0 MB 00:00 2024-06-14T23:58:57.893 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:58:57.893 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-06-14T23:58:57.956 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:58:57.971 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:58:57.971 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:58:58.057 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:58:58.057 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:58:58.576 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:58:58.721 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T23:58:58.747 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-14T23:58:58.809 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-14T23:58:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-14T23:58:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-14T23:58:58.943 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-14T23:58:59.037 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-14T23:58:59.138 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-14T23:58:59.194 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-14T23:58:59.809 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-14T23:58:59.837 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T23:58:59.915 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-14T23:58:59.967 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-14T23:59:00.000 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-14T23:59:00.047 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-14T23:59:00.070 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T23:59:00.829 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T23:59:00.829 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-14T23:59:00.830 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-14T23:59:00.830 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-14T23:59:00.830 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-14T23:59:00.830 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-14T23:59:00.830 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-14T23:59:00.831 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-14T23:59:01.155 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:01.156 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:01.406 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-06-14T23:59:01.987 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:20 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:02.114 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-14T23:59:02.115 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 7.0 M 2024-06-14T23:59:02.116 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-06-14T23:59:02.117 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:02.277 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-14T23:59:02.278 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-14T23:59:02.279 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-14T23:59:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-14T23:59:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:02.666 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:59:02.666 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-14T23:59:02.667 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:02.668 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:02.718 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 887 kB/s | 248 kB 00:00 2024-06-14T23:59:02.777 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 364 kB/s | 123 kB 00:00 2024-06-14T23:59:02.835 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.6 MB/s | 548 kB 00:00 2024-06-14T23:59:02.861 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.2 MB/s | 51 kB 00:00 2024-06-14T23:59:02.885 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 531 kB/s | 58 kB 00:00 2024-06-14T23:59:02.973 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-rook 2024-06-14T23:59:03.581 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:19 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:03.718 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-14T23:59:03.720 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout:Install 13 Packages 2024-06-14T23:59:03.721 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:03.722 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.2 M 2024-06-14T23:59:03.722 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 26 M 2024-06-14T23:59:03.722 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:04.111 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.6 MB/s | 6.0 MB 00:01 2024-06-14T23:59:04.113 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:04.113 INFO:teuthology.orchestra.run.smithi136.stdout:Total 3.5 MB/s | 7.0 MB 00:01 2024-06-14T23:59:04.184 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:04.195 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:04.195 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:04.292 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:04.292 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:04.304 INFO:teuthology.orchestra.run.smithi063.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 231 kB/s | 26 kB 00:00 2024-06-14T23:59:04.329 INFO:teuthology.orchestra.run.smithi063.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 141 kB/s | 19 kB 00:00 2024-06-14T23:59:04.379 INFO:teuthology.orchestra.run.smithi063.stdout:(3/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 254 kB/s | 47 kB 00:00 2024-06-14T23:59:04.413 INFO:teuthology.orchestra.run.smithi063.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-06-14T23:59:04.463 INFO:teuthology.orchestra.run.smithi063.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-14T23:59:04.480 INFO:teuthology.orchestra.run.smithi063.stdout:(6/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-06-14T23:59:04.555 INFO:teuthology.orchestra.run.smithi063.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 350 kB/s | 32 kB 00:00 2024-06-14T23:59:04.581 INFO:teuthology.orchestra.run.smithi063.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-06-14T23:59:04.606 INFO:teuthology.orchestra.run.smithi063.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 119 kB/s | 15 kB 00:00 2024-06-14T23:59:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:04.668 INFO:teuthology.orchestra.run.smithi063.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.6 MB/s | 59 kB 00:00 2024-06-14T23:59:04.693 INFO:teuthology.orchestra.run.smithi063.stdout:(11/13): python3-google-auth-2.29.0-2.el9.noarc 1.5 MB/s | 205 kB 00:00 2024-06-14T23:59:04.718 INFO:teuthology.orchestra.run.smithi063.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-06-14T23:59:04.760 INFO:teuthology.orchestra.run.smithi063.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.7 MB/s | 1.0 MB 00:00 2024-06-14T23:59:04.763 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:04.763 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-06-14T23:59:04.767 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-14T23:59:04.821 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-14T23:59:04.828 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:04.844 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:04.844 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:04.940 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:04.940 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:05.352 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-14T23:59:05.441 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-14T23:59:05.450 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:05.496 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-14T23:59:05.533 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T23:59:05.630 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-14T23:59:05.685 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-14T23:59:05.741 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-14T23:59:05.776 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-14T23:59:05.818 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-14T23:59:05.917 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-14T23:59:06.035 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-14T23:59:06.087 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-14T23:59:06.335 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-14T23:59:06.636 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-14T23:59:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-14T23:59:06.637 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:06.637 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:06.849 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-14T23:59:06.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-06-14T23:59:06.899 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-14T23:59:06.933 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-14T23:59:06.979 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-14T23:59:07.010 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T23:59:07.458 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:26 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:07.583 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:07.584 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-06-14T23:59:07.585 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:07.585 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 892 k 2024-06-14T23:59:07.585 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.6 M 2024-06-14T23:59:07.585 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-14T23:59:07.759 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-14T23:59:07.760 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-14T23:59:07.771 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 686 kB/s | 80 kB 00:00 2024-06-14T23:59:07.971 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 2.5 MB/s | 812 kB 00:00 2024-06-14T23:59:07.972 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:07.972 INFO:teuthology.orchestra.run.smithi136.stdout:Total 2.2 MB/s | 892 kB 00:00 2024-06-14T23:59:07.984 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:08.005 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:08.006 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:08.080 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-14T23:59:08.080 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:08.080 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:08.080 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:08.081 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:08.085 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:08.085 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:08.321 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-mgr-cephadm 2024-06-14T23:59:08.408 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-14T23:59:08.479 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T23:59:08.926 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:24 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:09.057 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:09.058 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:09.058 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:59:09.058 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:09.058 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:09.058 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout:Install 6 Packages 2024-06-14T23:59:09.059 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:09.060 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 7.0 M 2024-06-14T23:59:09.060 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 33 M 2024-06-14T23:59:09.060 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:09.151 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T23:59:09.151 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-14T23:59:09.394 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-14T23:59:09.394 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:09.394 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:09.395 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-14T23:59:09.395 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:09.395 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:09.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-06-14T23:59:09.732 INFO:teuthology.orchestra.run.smithi063.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-06-14T23:59:09.799 INFO:teuthology.orchestra.run.smithi063.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.2 MB/s | 548 kB 00:00 2024-06-14T23:59:09.825 INFO:teuthology.orchestra.run.smithi063.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-14T23:59:09.850 INFO:teuthology.orchestra.run.smithi063.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-14T23:59:09.933 INFO:teuthology.orchestra.run.smithi063.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-06-14T23:59:09.992 INFO:teuthology.orchestra.run.smithi063.stdout:(6/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 320 kB/s | 123 kB 00:00 2024-06-14T23:59:09.993 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:09.994 INFO:teuthology.orchestra.run.smithi063.stdout:Total 7.5 MB/s | 7.0 MB 00:00 2024-06-14T23:59:10.067 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:10.079 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:10.079 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:10.182 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:10.184 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:10.245 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:29 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:10.375 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:10.375 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 126 k 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 448 k 2024-06-14T23:59:10.376 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:10.490 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:10.594 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 581 kB/s | 126 kB 00:00 2024-06-14T23:59:10.594 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:10.599 INFO:teuthology.orchestra.run.smithi136.stdout:Total 577 kB/s | 126 kB 00:00 2024-06-14T23:59:10.599 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:10.600 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:10.600 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:10.662 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:10.662 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:10.674 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-14T23:59:10.719 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-14T23:59:10.814 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:10.924 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:11.270 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-14T23:59:11.349 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-14T23:59:11.382 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:11.414 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-14T23:59:11.439 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T23:59:11.643 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:11.644 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:11.644 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:11.644 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:11.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-14T23:59:12.188 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-14T23:59:12.451 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:31 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:12.538 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:12.581 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:12.582 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:12.582 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-14T23:59:12.595 INFO:teuthology.orchestra.run.smithi063.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-14T23:59:12.596 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:12.596 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:12.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-06-14T23:59:12.807 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install ceph-fuse 2024-06-14T23:59:13.233 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:32 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:13.358 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:13.359 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:59:13.360 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:13.360 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 25 k 2024-06-14T23:59:13.360 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 110 k 2024-06-14T23:59:13.360 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:13.407 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:29 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:13.539 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:13.540 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:13.540 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repository Size 2024-06-14T23:59:13.540 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:13.540 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:13.540 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout:Installing dependencies: 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout:Install 2 Packages 2024-06-14T23:59:13.541 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:13.542 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 892 k 2024-06-14T23:59:13.542 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 2.6 M 2024-06-14T23:59:13.542 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:13.607 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 101 kB/s | 25 kB 00:00 2024-06-14T23:59:13.607 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:13.608 INFO:teuthology.orchestra.run.smithi136.stdout:Total 100 kB/s | 25 kB 00:00 2024-06-14T23:59:13.608 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:13.611 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:13.611 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:13.612 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:13.613 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:13.694 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:13.751 INFO:teuthology.orchestra.run.smithi063.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 468 kB/s | 80 kB 00:00 2024-06-14T23:59:13.796 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:13.868 INFO:teuthology.orchestra.run.smithi063.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 2.7 MB/s | 812 kB 00:00 2024-06-14T23:59:13.868 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:13.869 INFO:teuthology.orchestra.run.smithi063.stdout:Total 2.7 MB/s | 892 kB 00:00 2024-06-14T23:59:13.881 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:13.903 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:13.903 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:13.985 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:13.985 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:14.181 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:14.187 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:14.344 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:14.409 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:14.411 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T23:59:14.615 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-06-14T23:59:15.056 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-14T23:59:15.056 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-14T23:59:15.193 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:15.277 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:15.318 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:15.319 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:15.319 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:15.328 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-14T23:59:15.328 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:15.328 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:15.329 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-14T23:59:15.329 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:15.329 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:15.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-06-14T23:59:15.581 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados-devel 2024-06-14T23:59:15.970 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:16.054 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:16.096 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:16.097 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:16.097 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:16.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-06-14T23:59:16.182 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:32 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:16.316 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:59:16.317 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:16.318 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 126 k 2024-06-14T23:59:16.318 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 448 k 2024-06-14T23:59:16.318 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:16.556 INFO:teuthology.orchestra.run.smithi063.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 530 kB/s | 126 kB 00:00 2024-06-14T23:59:16.557 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:16.557 INFO:teuthology.orchestra.run.smithi063.stdout:Total 527 kB/s | 126 kB 00:00 2024-06-14T23:59:16.557 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:16.563 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:16.563 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:16.629 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:16.629 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:16.749 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:35 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:16.782 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:16.834 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:16.876 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:16.877 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:16.877 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:16.881 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:16.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-06-14T23:59:17.273 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:17.530 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:36 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:17.547 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:17.547 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:17.549 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:17.613 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:17.654 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:17.655 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:17.656 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:17.740 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-06-14T23:59:17.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs2 2024-06-14T23:59:18.317 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:18.359 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:34 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:18.400 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:18.441 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:18.442 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:18.442 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:18.446 INFO:teuthology.orchestra.run.smithi063.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:18.491 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:18.492 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:18.492 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:18.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-06-14T23:59:18.557 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install libcephfs-devel 2024-06-14T23:59:19.094 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:38 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:19.158 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:35 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:19.180 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:19.221 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:19.222 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-06-14T23:59:19.222 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:19.289 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:19.290 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:19.290 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-06-14T23:59:19.290 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:19.290 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:19.290 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 25 k 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 110 k 2024-06-14T23:59:19.291 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:19.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-06-14T23:59:19.535 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 103 kB/s | 25 kB 00:00 2024-06-14T23:59:19.535 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:19.535 INFO:teuthology.orchestra.run.smithi063.stdout:Total 102 kB/s | 25 kB 00:00 2024-06-14T23:59:19.535 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:19.538 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:19.538 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:19.540 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:19.540 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:19.617 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:19.725 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:19.870 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:38 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:19.994 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:19.994 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:19.994 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-06-14T23:59:19.994 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:59:19.995 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:19.996 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 82 k 2024-06-14T23:59:19.996 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 222 k 2024-06-14T23:59:19.996 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:20.098 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:20.201 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 399 kB/s | 82 kB 00:00 2024-06-14T23:59:20.202 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:20.202 INFO:teuthology.orchestra.run.smithi136.stdout:Total 396 kB/s | 82 kB 00:00 2024-06-14T23:59:20.202 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:20.209 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:20.209 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:20.271 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:20.271 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:20.345 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:20.415 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:20.506 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:20.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librados2 2024-06-14T23:59:20.876 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:21.117 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:21.150 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:21.205 INFO:teuthology.orchestra.run.smithi063.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:21.249 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:21.250 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:21.250 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:21.315 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install librbd1 2024-06-14T23:59:21.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-06-14T23:59:21.907 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:37 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:21.975 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:40 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:21.993 INFO:teuthology.orchestra.run.smithi063.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:22.037 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:22.038 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:22.038 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:22.103 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rados 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.9 M 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 12 M 2024-06-14T23:59:22.104 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:22.472 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 8.0 MB/s | 2.9 MB 00:00 2024-06-14T23:59:22.473 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:22.474 INFO:teuthology.orchestra.run.smithi136.stdout:Total 7.9 MB/s | 2.9 MB 00:00 2024-06-14T23:59:22.474 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:22.482 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:22.483 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:22.561 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:22.562 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:22.710 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:38 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:22.796 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:22.840 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:22.842 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:22.842 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:22.906 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rgw 2024-06-14T23:59:22.973 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:23.023 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T23:59:23.047 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:23.507 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:39 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:23.594 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:23.638 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:23.639 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:23.639 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:23.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-cephfs 2024-06-14T23:59:24.084 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:24.085 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:24.085 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:24.085 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:24.294 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-06-14T23:59:24.301 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:40 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:24.389 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:24.497 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install python3-rbd 2024-06-14T23:59:24.888 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:43 ago on Fri 14 Jun 2024 11:57:41 PM UTC. 2024-06-14T23:59:25.014 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-06-14T23:59:25.015 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-06-14T23:59:25.016 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-06-14T23:59:25.016 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:25.016 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 165 k 2024-06-14T23:59:25.016 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 477 k 2024-06-14T23:59:25.016 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-06-14T23:59:25.099 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:41 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:25.187 INFO:teuthology.orchestra.run.smithi063.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-14T23:59:25.231 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:25.232 INFO:teuthology.orchestra.run.smithi063.stdout:Nothing to do. 2024-06-14T23:59:25.232 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:25.242 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 728 kB/s | 165 kB 00:00 2024-06-14T23:59:25.243 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:25.243 INFO:teuthology.orchestra.run.smithi136.stdout:Total 723 kB/s | 165 kB 00:00 2024-06-14T23:59:25.243 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-06-14T23:59:25.251 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-06-14T23:59:25.251 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-06-14T23:59:25.297 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-fuse 2024-06-14T23:59:25.315 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-06-14T23:59:25.315 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-06-14T23:59:25.464 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-06-14T23:59:25.565 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:25.896 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:41 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:26.027 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:26.028 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:26.028 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:59:26.029 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:26.030 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 82 k 2024-06-14T23:59:26.030 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 222 k 2024-06-14T23:59:26.030 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:26.251 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:26.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:26.260 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-06-14T23:59:26.260 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:26.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-14T23:59:26.260 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-06-14T23:59:26.295 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 308 kB/s | 82 kB 00:00 2024-06-14T23:59:26.296 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:26.296 INFO:teuthology.orchestra.run.smithi063.stdout:Total 306 kB/s | 82 kB 00:00 2024-06-14T23:59:26.296 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:26.303 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:26.303 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:26.368 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:26.369 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:26.475 DEBUG:teuthology.parallel:result is None 2024-06-14T23:59:26.516 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:26.612 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:26.976 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:27.235 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:27.236 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:27.236 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:27.236 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:27.236 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:27.236 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:27.460 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-mirror 2024-06-14T23:59:28.066 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:44 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:28.199 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout: Package Arch Version Repo Size 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:28.200 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 2.9 M 2024-06-14T23:59:28.201 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 12 M 2024-06-14T23:59:28.201 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:28.590 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 7.6 MB/s | 2.9 MB 00:00 2024-06-14T23:59:28.590 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:28.591 INFO:teuthology.orchestra.run.smithi063.stdout:Total 7.5 MB/s | 2.9 MB 00:00 2024-06-14T23:59:28.591 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:28.601 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:28.601 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:28.683 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:28.684 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:29.117 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:29.165 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:29.191 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:29.192 INFO:teuthology.orchestra.run.smithi063.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-14T23:59:29.192 INFO:teuthology.orchestra.run.smithi063.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-14T23:59:29.192 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T23:59:29.192 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-14T23:59:29.192 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:30.311 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:30.520 DEBUG:teuthology.orchestra.run.smithi063:> sudo yum -y install rbd-nbd 2024-06-14T23:59:31.127 INFO:teuthology.orchestra.run.smithi063.stdout:Last metadata expiration check: 0:01:47 ago on Fri 14 Jun 2024 11:57:44 PM UTC. 2024-06-14T23:59:31.260 INFO:teuthology.orchestra.run.smithi063.stdout:Dependencies resolved. 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: Package Architecture Version Repository Size 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:Installing: 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction Summary 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:================================================================================ 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:Install 1 Package 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:31.261 INFO:teuthology.orchestra.run.smithi063.stdout:Total download size: 165 k 2024-06-14T23:59:31.262 INFO:teuthology.orchestra.run.smithi063.stdout:Installed size: 477 k 2024-06-14T23:59:31.262 INFO:teuthology.orchestra.run.smithi063.stdout:Downloading Packages: 2024-06-14T23:59:31.609 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 475 kB/s | 165 kB 00:00 2024-06-14T23:59:31.610 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-14T23:59:31.610 INFO:teuthology.orchestra.run.smithi063.stdout:Total 473 kB/s | 165 kB 00:00 2024-06-14T23:59:31.610 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction check 2024-06-14T23:59:31.618 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction check succeeded. 2024-06-14T23:59:31.618 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction test 2024-06-14T23:59:31.684 INFO:teuthology.orchestra.run.smithi063.stdout:Transaction test succeeded. 2024-06-14T23:59:31.684 INFO:teuthology.orchestra.run.smithi063.stdout:Running transaction 2024-06-14T23:59:31.842 INFO:teuthology.orchestra.run.smithi063.stdout: Preparing : 1/1 2024-06-14T23:59:31.943 INFO:teuthology.orchestra.run.smithi063.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:32.338 INFO:teuthology.orchestra.run.smithi063.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout:Installed: 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-14T23:59:32.572 INFO:teuthology.orchestra.run.smithi063.stdout:Complete! 2024-06-14T23:59:32.821 DEBUG:teuthology.parallel:result is None 2024-06-14T23:59:32.822 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:59:32.822 INFO:teuthology.packaging:ref: None 2024-06-14T23:59:32.822 INFO:teuthology.packaging:tag: None 2024-06-14T23:59:32.822 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:59:32.822 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:59:32.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T23:59:33.028 DEBUG:teuthology.orchestra.run.smithi063:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-14T23:59:33.050 INFO:teuthology.orchestra.run.smithi063.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-14T23:59:33.051 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-14T23:59:33.051 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-14T23:59:33.053 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-14T23:59:33.053 INFO:teuthology.packaging:ref: None 2024-06-14T23:59:33.053 INFO:teuthology.packaging:tag: None 2024-06-14T23:59:33.053 INFO:teuthology.packaging:branch: quincy 2024-06-14T23:59:33.053 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-14T23:59:33.053 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-14T23:59:33.343 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-14T23:59:33.369 INFO:teuthology.orchestra.run.smithi136.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-14T23:59:33.370 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-14T23:59:33.370 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-14T23:59:33.371 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-14T23:59:33.372 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:59:33.372 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-14T23:59:33.398 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:59:33.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-14T23:59:33.440 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-14T23:59:33.441 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:59:33.441 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2024-06-14T23:59:33.468 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-14T23:59:33.541 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:59:33.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-06-14T23:59:33.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-14T23:59:33.638 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-14T23:59:33.638 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:59:33.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-14T23:59:33.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-14T23:59:33.730 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:59:33.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-14T23:59:33.757 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-14T23:59:33.823 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-14T23:59:33.823 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-14T23:59:33.824 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/stdin-killer 2024-06-14T23:59:33.849 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-14T23:59:33.915 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-14T23:59:33.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-06-14T23:59:33.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-14T23:59:34.007 INFO:teuthology.run_tasks:Running task print... 2024-06-14T23:59:34.016 INFO:teuthology.task.print:**** done install task... 2024-06-14T23:59:34.016 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-14T23:59:34.126 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-14T23:59:34.127 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-14T23:59:34.127 INFO:tasks.cephadm:Cluster fsid is 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-14T23:59:34.127 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-14T23:59:34.127 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-14T23:59:34.128 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi063': '172.21.15.63', 'mon.smithi136': '172.21.15.136'} 2024-06-14T23:59:34.128 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-14T23:59:34.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo hostname $(hostname -s) 2024-06-14T23:59:34.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo hostname $(hostname -s) 2024-06-14T23:59:34.179 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-14T23:59:34.179 DEBUG:teuthology.orchestra.run.smithi063:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.405 INFO:teuthology.orchestra.run.smithi063.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 14 23:59 /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.405 DEBUG:teuthology.orchestra.run.smithi136:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.584 INFO:teuthology.orchestra.run.smithi136.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 14 23:59 /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.584 DEBUG:teuthology.orchestra.run.smithi063:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.601 DEBUG:teuthology.orchestra.run.smithi136:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-14T23:59:34.626 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-14T23:59:34.626 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-14T23:59:34.644 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-14T23:59:35.524 INFO:teuthology.orchestra.run.smithi063.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-14T23:59:35.595 INFO:teuthology.orchestra.run.smithi136.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:00:17.376 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-06-15T00:00:17.376 INFO:teuthology.orchestra.run.smithi136.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-15T00:00:17.376 INFO:teuthology.orchestra.run.smithi136.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-15T00:00:17.376 INFO:teuthology.orchestra.run.smithi136.stdout: "repo_digests": [ 2024-06-15T00:00:17.377 INFO:teuthology.orchestra.run.smithi136.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-15T00:00:17.377 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-06-15T00:00:17.377 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-06-15T00:00:20.998 INFO:teuthology.orchestra.run.smithi063.stdout:{ 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout: "repo_digests": [ 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout: ] 2024-06-15T00:00:20.999 INFO:teuthology.orchestra.run.smithi063.stdout:} 2024-06-15T00:00:21.019 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph 2024-06-15T00:00:21.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph 2024-06-15T00:00:21.094 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /etc/ceph 2024-06-15T00:00:21.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /etc/ceph 2024-06-15T00:00:21.162 INFO:tasks.cephadm:Writing seed config... 2024-06-15T00:00:21.163 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-15T00:00:21.163 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-15T00:00:21.164 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-15T00:00:21.165 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-15T00:00:21.166 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-15T00:00:21.167 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-15T00:00:21.168 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-15T00:00:21.168 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-15T00:00:21.168 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-15T00:00:21.168 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-15T00:00:21.168 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-15T00:00:21.169 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:00:21.169 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-15T00:00:21.185 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 25f32e20-2aaa-11ef-bc9d-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-15T00:00:21.185 DEBUG:teuthology.orchestra.run.smithi063:mon.smithi063> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service 2024-06-15T00:00:21.228 INFO:tasks.cephadm:Bootstrapping... 2024-06-15T00:00:21.228 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.63 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:00:21.444 INFO:teuthology.orchestra.run.smithi063.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:00:21.445 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '25f32e20-2aaa-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.63', '--skip-admin-label'] 2024-06-15T00:00:21.470 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:00:21.471 INFO:teuthology.orchestra.run.smithi063.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-15T00:00:21.471 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying podman|docker is present... 2024-06-15T00:00:21.496 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:00:21.496 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying lvm2 is present... 2024-06-15T00:00:21.496 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying time synchronization is in place... 2024-06-15T00:00:21.504 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-15T00:00:21.504 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-15T00:00:21.511 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-15T00:00:21.511 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:00:21.518 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-06-15T00:00:21.525 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-06-15T00:00:21.525 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-06-15T00:00:21.525 INFO:teuthology.orchestra.run.smithi063.stdout:Repeating the final host check... 2024-06-15T00:00:21.549 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:00:21.549 INFO:teuthology.orchestra.run.smithi063.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-15T00:00:21.549 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl is present 2024-06-15T00:00:21.549 INFO:teuthology.orchestra.run.smithi063.stdout:lvcreate is present 2024-06-15T00:00:21.556 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-15T00:00:21.556 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-15T00:00:21.562 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-15T00:00:21.563 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:00:21.570 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout enabled 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout active 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:Unit chronyd.service is enabled and running 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:Host looks OK 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:Cluster fsid: 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:Acquiring lock 140108189843904 on /run/cephadm/25f32e20-2aaa-11ef-bc9d-c7b262605968.lock 2024-06-15T00:00:21.577 INFO:teuthology.orchestra.run.smithi063.stdout:Lock 140108189843904 acquired on /run/cephadm/25f32e20-2aaa-11ef-bc9d-c7b262605968.lock 2024-06-15T00:00:21.578 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 3300 ... 2024-06-15T00:00:21.578 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying IP 172.21.15.63 port 6789 ... 2024-06-15T00:00:21.578 INFO:teuthology.orchestra.run.smithi063.stdout:Base mon IP(s) is [172.21.15.63:3300, 172.21.15.63:6789], mon addrv is [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-06-15T00:00:21.582 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.63 metric 100 2024-06-15T00:00:21.582 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.63 metric 100 2024-06-15T00:00:21.585 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-15T00:00:21.585 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-15T00:00:21.585 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-06-15T00:00:21.587 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:feb6:f8d7/64 scope link 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-15T00:00:21.588 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-06-15T00:00:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Mon IP `172.21.15.63` is in CIDR network `172.21.0.0/20` 2024-06-15T00:00:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-15T00:00:21.589 INFO:teuthology.orchestra.run.smithi063.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-15T00:00:21.590 INFO:teuthology.orchestra.run.smithi063.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:00:22.697 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stdout 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Getting image source signatures 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:ae1bd4835d962ece7feffb132e988e26ca074ae51fc4b41b3d8d10a6caf0861f 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Copying config sha256:37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-15T00:00:22.698 INFO:teuthology.orchestra.run.smithi063.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-15T00:00:23.560 INFO:teuthology.orchestra.run.smithi063.stdout:ceph: stdout ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-15T00:00:23.561 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph version: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-15T00:00:23.561 INFO:teuthology.orchestra.run.smithi063.stdout:Extracting ceph user uid/gid from container image... 2024-06-15T00:00:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:stat: stdout 167 167 2024-06-15T00:00:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial keys... 2024-06-15T00:00:25.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCY2WxmGt0TLBAAG1cpPPuBOv8jf8J1sIO2gw== 2024-06-15T00:00:26.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCZ2Wxmcp/wIhAA60tem5x4bEwSqImKIbZsaw== 2024-06-15T00:00:27.056 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph-authtool: stdout AQCa2Wxmu0XOHxAAzxDHZlVOX0avh5wNrn8s7g== 2024-06-15T00:00:27.057 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial monmap... 2024-06-15T00:00:27.978 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool for smithi063 [v2:172.21.15.63:3300,v1:172.21.15.63:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:setting min_mon_release = octopus 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: set fsid to 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:00:27.979 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mon... 2024-06-15T00:00:28.893 INFO:teuthology.orchestra.run.smithi063.stdout:create mon.smithi063 on 2024-06-15T00:00:29.112 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-15T00:00:29.292 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-15T00:00:29.466 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target. 2024-06-15T00:00:29.466 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target. 2024-06-15T00:00:29.669 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063 2024-06-15T00:00:29.669 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service: Unit ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service not loaded. 2024-06-15T00:00:29.843 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target.wants/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service → /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@.service. 2024-06-15T00:00:29.902 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:29 smithi063 systemd[1]: Starting Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:00:30.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 podman[28501]: 2024-06-15 00:00:30.084250206 +0000 UTC m=+0.105673094 container create a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-06-15T00:00:30.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 podman[28501]: 2024-06-15 00:00:29.997305404 +0000 UTC m=+0.018728306 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-15T00:00:30.361 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:00:30.361 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-06-15T00:00:30.369 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:00:30.369 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:00:30.369 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-06-15T00:00:30.369 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-06-15T00:00:30.370 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon to start... 2024-06-15T00:00:30.370 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mon... 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 podman[28501]: 2024-06-15 00:00:30.261333739 +0000 UTC m=+0.282756627 container init a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, org.label-schema.build-date=20240612, ceph=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 podman[28501]: 2024-06-15 00:00:30.264556117 +0000 UTC m=+0.285979014 container start a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 ceph-mon[28520]: mkfs 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 ceph-mon[28520]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 bash[28501]: a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e 2024-06-15T00:00:30.473 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:30 smithi063 systemd[1]: Started Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968. 2024-06-15T00:00:31.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-06-15T00:00:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: monmap e1: 1 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:00:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: fsmap 2024-06-15T00:00:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: osdmap e1: 0 total, 0 up, 0 in 2024-06-15T00:00:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: mgrmap e1: no daemons active 2024-06-15T00:00:31.383 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:31 smithi063 ceph-mon[28520]: from='client.? 172.21.15.63:0/2975230459' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-15T00:00:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout cluster: 2024-06-15T00:00:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout id: 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-15T00:00:31.390 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout services: 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi063 (age 0.539431s) 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout data: 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout pgs: 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.765+0000 7fdc438e8640 1 Processor -- start 2024-06-15T00:00:31.391 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc438e8640 1 -- start start 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc438e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc438e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c1012e0 con 0x7fdc3c100990 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc41684640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc41684640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34658/0 (socket says 172.21.15.63:34658) 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.766+0000 7fdc41684640 1 -- 172.21.15.63:0/3749212302 learned_addr learned my addr 172.21.15.63:0/3749212302 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.767+0000 7fdc41684640 1 -- 172.21.15.63:0/3749212302 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc3c101420 con 0x7fdc3c100990 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc41684640 1 --2- 172.21.15.63:0/3749212302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009840 tx=0x7fdc380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfb57ec7ea7dc6e server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc2ffff640 1 -- 172.21.15.63:0/3749212302 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38032440 con 0x7fdc3c100990 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc2ffff640 1 -- 172.21.15.63:0/3749212302 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc380325a0 con 0x7fdc3c100990 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc2ffff640 1 -- 172.21.15.63:0/3749212302 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc380328d0 con 0x7fdc3c100990 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc438e8640 1 -- 172.21.15.63:0/3749212302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 msgr2=0x7fdc3c100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:31.392 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc438e8640 1 --2- 172.21.15.63:0/3749212302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fdc38009840 tx=0x7fdc380091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc438e8640 1 -- 172.21.15.63:0/3749212302 shutdown_connections 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc438e8640 1 --2- 172.21.15.63:0/3749212302 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c100da0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.769+0000 7fdc438e8640 1 -- 172.21.15.63:0/3749212302 >> 172.21.15.63:0/3749212302 conn(0x7fdc3c0fc150 msgr2=0x7fdc3c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 -- 172.21.15.63:0/3749212302 shutdown_connections 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 -- 172.21.15.63:0/3749212302 wait complete. 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 Processor -- start 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 -- start start 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc438e8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c113030 con 0x7fdc3c100990 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc41684640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc41684640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34660/0 (socket says 172.21.15.63:34660) 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.770+0000 7fdc41684640 1 -- 172.21.15.63:0/2975230459 learned_addr learned my addr 172.21.15.63:0/2975230459 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc41684640 1 -- 172.21.15.63:0/2975230459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc38009450 con 0x7fdc3c100990 2024-06-15T00:00:31.393 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc41684640 1 --2- 172.21.15.63:0/2975230459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035040 tx=0x7fdc38032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc38031b60 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc38031cc0 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c113170 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc3803b400 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c113580 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fdc38039040 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.771+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdc38041070 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.772+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c04b480 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.774+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fdc3804a060 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.838+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fdc3c063f70 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.838+0000 7fdc2dffb640 1 -- 172.21.15.63:0/2975230459 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fdc38037070 con 0x7fdc3c100990 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 msgr2=0x7fdc3c075b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:31.394 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 --2- 172.21.15.63:0/2975230459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fdc38035040 tx=0x7fdc38032000 comp rx=0 tx=0).stop 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 shutdown_connections 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 --2- 172.21.15.63:0/2975230459 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdc3c100990 0x7fdc3c075b90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 >> 172.21.15.63:0/2975230459 conn(0x7fdc3c0fc150 msgr2=0x7fdc3c105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 shutdown_connections 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:30.839+0000 7fdc438e8640 1 -- 172.21.15.63:0/2975230459 wait complete. 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:mon is available 2024-06-15T00:00:31.395 INFO:teuthology.orchestra.run.smithi063.stdout:Assimilating anything we can from ceph.conf... 2024-06-15T00:00:32.422 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:32 smithi063 ceph-mon[28520]: from='client.? 172.21.15.63:0/780179774' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-15T00:00:32.422 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:32 smithi063 ceph-mon[28520]: from='client.? 172.21.15.63:0/780179774' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.63:3300,v1:172.21.15.63:6789] 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-15T00:00:32.436 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-06-15T00:00:32.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.781+0000 7f061010c640 1 Processor -- start 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.781+0000 7f061010c640 1 -- start start 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.781+0000 7f061010c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:32.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.781+0000 7f061010c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06081018f0 con 0x7f0608100fa0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.782+0000 7f060dea8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.782+0000 7f060dea8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34672/0 (socket says 172.21.15.63:34672) 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.782+0000 7f060dea8640 1 -- 172.21.15.63:0/459562822 learned_addr learned my addr 172.21.15.63:0/459562822 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.782+0000 7f060dea8640 1 -- 172.21.15.63:0/459562822 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608101a30 con 0x7f0608100fa0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f060dea8640 1 --2- 172.21.15.63:0/459562822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f060400baf0 tx=0x7f060400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9bb96000dfe174b server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f060cea6640 1 -- 172.21.15.63:0/459562822 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06040327a0 con 0x7f0608100fa0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f060cea6640 1 -- 172.21.15.63:0/459562822 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0604032900 con 0x7f0608100fa0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f060cea6640 1 -- 172.21.15.63:0/459562822 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0604032c30 con 0x7f0608100fa0 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f061010c640 1 -- 172.21.15.63:0/459562822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 msgr2=0x7f06081013b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:32.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f061010c640 1 --2- 172.21.15.63:0/459562822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f060400baf0 tx=0x7f060400b460 comp rx=0 tx=0).stop 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f061010c640 1 -- 172.21.15.63:0/459562822 shutdown_connections 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f061010c640 1 --2- 172.21.15.63:0/459562822 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f06081013b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.783+0000 7f061010c640 1 -- 172.21.15.63:0/459562822 >> 172.21.15.63:0/459562822 conn(0x7f06080fb990 msgr2=0x7f06080fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 -- 172.21.15.63:0/459562822 shutdown_connections 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 -- 172.21.15.63:0/459562822 wait complete. 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 Processor -- start 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 -- start start 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f061010c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f060810e420 con 0x7f0608100fa0 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f060dea8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f060dea8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34676/0 (socket says 172.21.15.63:34676) 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.784+0000 7f060dea8640 1 -- 172.21.15.63:0/780179774 learned_addr learned my addr 172.21.15.63:0/780179774 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:32.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f060dea8640 1 -- 172.21.15.63:0/780179774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f060400b700 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f060dea8640 1 --2- 172.21.15.63:0/780179774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f060400baf0 tx=0x7f060403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060403c720 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f060403c880 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f060810e620 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f060403cb70 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f060810f0d0 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f0604045070 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.785+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f06040518d0 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.786+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f060805ead0 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.788+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f060404c070 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.849+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f060804b480 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.853+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f060403ccd0 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.853+0000 7f05fe7fc640 1 -- 172.21.15.63:0/780179774 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f0604036020 con 0x7f0608100fa0 2024-06-15T00:00:32.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 msgr2=0x7f060810dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 --2- 172.21.15.63:0/780179774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f060400baf0 tx=0x7f060403a000 comp rx=0 tx=0).stop 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 shutdown_connections 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 --2- 172.21.15.63:0/780179774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0608100fa0 0x7f060810dee0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 >> 172.21.15.63:0/780179774 conn(0x7f06080fb990 msgr2=0x7f06080fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 shutdown_connections 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:31.854+0000 7f061010c640 1 -- 172.21.15.63:0/780179774 wait complete. 2024-06-15T00:00:32.442 INFO:teuthology.orchestra.run.smithi063.stdout:Generating new minimal ceph.conf... 2024-06-15T00:00:33.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:33 smithi063 ceph-mon[28520]: from='client.? 172.21.15.63:0/3929648351' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:00:33.410 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.832+0000 7fd48f10e640 1 Processor -- start 2024-06-15T00:00:33.410 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.832+0000 7fd48f10e640 1 -- start start 2024-06-15T00:00:33.410 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.832+0000 7fd48f10e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.832+0000 7fd48f10e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881018b0 con 0x7fd488100f60 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.832+0000 7fd48ceaa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.833+0000 7fd48ceaa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34686/0 (socket says 172.21.15.63:34686) 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.833+0000 7fd48ceaa640 1 -- 172.21.15.63:0/2372222943 learned_addr learned my addr 172.21.15.63:0/2372222943 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.833+0000 7fd48ceaa640 1 -- 172.21.15.63:0/2372222943 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4881019f0 con 0x7fd488100f60 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48ceaa640 1 --2- 172.21.15.63:0/2372222943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd478009840 tx=0x7fd4780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a747385dce8fbf server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd4877fe640 1 -- 172.21.15.63:0/2372222943 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd478032440 con 0x7fd488100f60 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd4877fe640 1 -- 172.21.15.63:0/2372222943 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd478032a10 con 0x7fd488100f60 2024-06-15T00:00:33.411 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd4877fe640 1 -- 172.21.15.63:0/2372222943 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd478031a00 con 0x7fd488100f60 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 -- 172.21.15.63:0/2372222943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 msgr2=0x7fd488101370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 --2- 172.21.15.63:0/2372222943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd478009840 tx=0x7fd4780091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 -- 172.21.15.63:0/2372222943 shutdown_connections 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 --2- 172.21.15.63:0/2372222943 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd488101370 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 -- 172.21.15.63:0/2372222943 >> 172.21.15.63:0/2372222943 conn(0x7fd4880fb990 msgr2=0x7fd4880fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 -- 172.21.15.63:0/2372222943 shutdown_connections 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.834+0000 7fd48f10e640 1 -- 172.21.15.63:0/2372222943 wait complete. 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.835+0000 7fd48f10e640 1 Processor -- start 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.835+0000 7fd48f10e640 1 -- start start 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.835+0000 7fd48f10e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:33.412 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.835+0000 7fd48f10e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48810fcb0 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.835+0000 7fd48ceaa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48ceaa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34696/0 (socket says 172.21.15.63:34696) 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48ceaa640 1 -- 172.21.15.63:0/3929648351 learned_addr learned my addr 172.21.15.63:0/3929648351 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48ceaa640 1 -- 172.21.15.63:0/3929648351 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478009450 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48ceaa640 1 --2- 172.21.15.63:0/3929648351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd4780313e0 tx=0x7fd478008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd47803d450 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fd47803da20 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48810feb0 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd47803b8d0 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488110320 con 0x7fd488100f60 2024-06-15T00:00:33.413 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd478039070 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.836+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd478046700 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.837+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd48804b480 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.839+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fd488110320 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.901+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fd488102710 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd4857fa640 1 -- 172.21.15.63:0/3929648351 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7fd478041020 con 0x7fd488100f60 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 msgr2=0x7fd48810f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd48f10e640 1 --2- 172.21.15.63:0/3929648351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd4780313e0 tx=0x7fd478008040 comp rx=0 tx=0).stop 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 shutdown_connections 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd48f10e640 1 --2- 172.21.15.63:0/3929648351 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd488100f60 0x7fd48810f770 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:33.414 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.902+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 >> 172.21.15.63:0/3929648351 conn(0x7fd4880fb990 msgr2=0x7fd4880fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:33.415 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.903+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 shutdown_connections 2024-06-15T00:00:33.415 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:32.903+0000 7fd48f10e640 1 -- 172.21.15.63:0/3929648351 wait complete. 2024-06-15T00:00:33.415 INFO:teuthology.orchestra.run.smithi063.stdout:Restarting the monitor... 2024-06-15T00:00:33.674 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:33 smithi063 systemd[1]: Stopping Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:00:33.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:33 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063[28516]: 2024-06-15T00:00:33.673+0000 7faff02f7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi063 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:00:33.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:33 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063[28516]: 2024-06-15T00:00:33.673+0000 7faff02f7640 -1 mon.smithi063@0(leader) e1 *** Got Signal Terminated *** 2024-06-15T00:00:33.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:33 smithi063 podman[28762]: 2024-06-15 00:00:33.736122672 +0000 UTC m=+0.153728745 container died a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True) 2024-06-15T00:00:34.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28762]: 2024-06-15 00:00:34.082229319 +0000 UTC m=+0.499835397 container cleanup a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-06-15T00:00:34.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 bash[28762]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063 2024-06-15T00:00:34.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28774]: 2024-06-15 00:00:34.266361472 +0000 UTC m=+0.520400772 container remove a1bbab3640b5a94451b8121f58149848af4a99632de902219d2c0da4aee60b4e (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:00:34.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 systemd[1]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service: Deactivated successfully. 2024-06-15T00:00:34.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 systemd[1]: Stopped Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968. 2024-06-15T00:00:34.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 systemd[1]: Starting Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:00:34.922 INFO:teuthology.orchestra.run.smithi063.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28863]: 2024-06-15 00:00:34.69225496 +0000 UTC m=+0.104863850 container create 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28863]: 2024-06-15 00:00:34.604845881 +0000 UTC m=+0.017454788 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28863]: 2024-06-15 00:00:34.852486211 +0000 UTC m=+0.265095123 container init 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 podman[28863]: 2024-06-15 00:00:34.855583733 +0000 UTC m=+0.268192624 container start 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: set uid:gid to 167:167 (ceph:ceph) 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable), process ceph-mon, pid 2 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: pidfile_write: ignore empty --pid-file 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: load: jerasure load: lrc 2024-06-15T00:00:35.036 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: RocksDB version: 6.15.5 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Compile date Jun 14 2024 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: DB SUMMARY 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: DB Session ID: AZQALU2H48AB987XRLRH 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: CURRENT file: CURRENT 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: IDENTITY file: IDENTITY 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi063/store.db dir, Total Num: 1, files: 000008.sst 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi063/store.db: 000010.log size: 96776 ; 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.error_if_exists: 0 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.create_if_missing: 0 2024-06-15T00:00:35.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.paranoid_checks: 1 2024-06-15T00:00:35.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-15T00:00:35.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.env: 0x55c73cf9c440 2024-06-15T00:00:35.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.fs: Posix File System 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.info_log: 0x55c73e04b140 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_file_opening_threads: 16 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.statistics: (nil) 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.use_fsync: 0 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_log_file_size: 0 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.keep_log_file_num: 1000 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.recycle_log_file_num: 0 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_fallocate: 1 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_mmap_reads: 0 2024-06-15T00:00:35.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_mmap_writes: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.use_direct_reads: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.create_missing_column_families: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.db_log_dir: 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi063/store.db 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.advise_random_on_open: 1 2024-06-15T00:00:35.040 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.db_write_buffer_size: 0 2024-06-15T00:00:35.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.write_buffer_manager: 0x55c73e108840 2024-06-15T00:00:35.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-15T00:00:35.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.rate_limiter: (nil) 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.wal_recovery_mode: 2 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.enable_thread_tracking: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.enable_pipelined_write: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.unordered_write: 0 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-15T00:00:35.042 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.row_cache: None 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.wal_filter: None 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_ingest_behind: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.preserve_deletes: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.two_write_queues: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.manual_wal_flush: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.atomic_flush: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.log_readahead_size: 0 2024-06-15T00:00:35.043 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.best_efforts_recovery: 0 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.allow_data_in_errors: 0 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.db_host_id: __hostname__ 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_background_jobs: 2 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_background_compactions: -1 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_subcompactions: 1 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_total_wal_size: 0 2024-06-15T00:00:35.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_open_files: -1 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bytes_per_sync: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_readahead_size: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_background_flushes: -1 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Compression algorithms supported: 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kZSTD supported: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kXpressCompression supported: 0 2024-06-15T00:00:35.045 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kBZip2Compression supported: 0 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kLZ4Compression supported: 1 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kZlibCompression supported: 1 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kLZ4HCCompression supported: 1 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: kSnappyCompression supported: 1 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000009 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.merge_operator: 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_filter: None 2024-06-15T00:00:35.046 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_filter_factory: None 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.sst_partitioner_factory: None 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c73dff1a68) 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks: 1 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: pin_top_level_index_and_filter: 1 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_type: 0 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_index_type: 0 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_shortening: 1 2024-06-15T00:00:35.047 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: hash_index_allow_collision: 1 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: checksum: 1 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: no_block_cache: 0 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache: 0x55c73e0271f0 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_name: BinnedLRUCache 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_options: 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: capacity : 536870912 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: num_shard_bits : 4 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: strict_capacity_limit : 0 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: high_pri_pool_ratio: 0.000 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_cache_compressed: (nil) 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: persistent_cache: (nil) 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size: 4096 2024-06-15T00:00:35.048 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_size_deviation: 10 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_restart_interval: 16 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: index_block_restart_interval: 1 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: metadata_block_size: 4096 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: partition_filters: 0 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: use_delta_encoding: 1 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: whole_key_filtering: 1 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: verify_compression: 0 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: read_amp_bytes_per_bit: 0 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: format_version: 4 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_index_compression: 1 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: block_align: 0 2024-06-15T00:00:35.049 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.write_buffer_size: 33554432 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_write_buffer_number: 2 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression: NoCompression 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression: Disabled 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.prefix_extractor: nullptr 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.num_levels: 7 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-15T00:00:35.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.level: 32767 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.strategy: 0 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compression_opts.enabled: false 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.target_file_size_base: 67108864 2024-06-15T00:00:35.051 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-15T00:00:35.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.arena_block_size: 4194304 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-15T00:00:35.053 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.disable_auto_compactions: 0 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.table_properties_collectors: 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.inplace_update_support: 0 2024-06-15T00:00:35.054 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.bloom_locality: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.max_successive_merges: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.paranoid_file_checks: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.force_consistency_checks: 1 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.report_bg_io_stats: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.ttl: 2592000 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.enable_blob_files: false 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.min_blob_size: 0 2024-06-15T00:00:35.055 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.blob_file_size: 268435456 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi063/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718409634879188, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718409634880442, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92145, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90240, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 82979, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718409634, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "26057b69-2bab-48fc-bdb4-cc51551c050d", "db_session_id": "AZQALU2H48AB987XRLRH"}} 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718409634880934, "job": 1, "event": "recovery_finished"} 2024-06-15T00:00:35.056 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi063/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55c73e05a700 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: DB pointer 0x55c73e02a000 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** DB Stats ** 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:00:35.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L0 2/0 91.78 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Sum 2/0 91.78 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.058 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative compaction: 0.00 GB write, 20.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval compaction: 0.00 GB write, 20.24 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:00:35.059 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: L0 2/0 91.78 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Sum 2/0 91.78 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** Compaction Stats [default] ** 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 108.4 0.00 0.00 1 0.001 0 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:00:35.060 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Cumulative compaction: 0.00 GB write, 20.09 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: starting mon.smithi063 rank 0 at public addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] at bind addrs [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi063 fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???) e1 preinit fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).mds e1 new map 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).mds e1 print_map 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: e1 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: legacy client fscid: -1 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: 2024-06-15T00:00:35.061 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: No filesystems configured 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 bash[28863]: 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 systemd[1]: Started Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968. 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mon.smithi063 is new leader, mons smithi063 in quorum (ranks 0) 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: monmap e1: 1 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: fsmap 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: osdmap e1: 0 total, 0 up, 0 in 2024-06-15T00:00:35.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:34 smithi063 ceph-mon[28882]: mgrmap e1: no daemons active 2024-06-15T00:00:36.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:36 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2607107145' entity='client.admin' 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.336+0000 7f8ac2091640 1 Processor -- start 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.336+0000 7f8ac2091640 1 -- start start 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.336+0000 7f8ac2091640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.336+0000 7f8ac2091640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc0780a0 con 0x7f8abc077c90 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.337+0000 7f8abb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.337+0000 7f8abb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34700/0 (socket says 172.21.15.63:34700) 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.337+0000 7f8abb7fe640 1 -- 172.21.15.63:0/3009714365 learned_addr learned my addr 172.21.15.63:0/3009714365 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.337+0000 7f8abb7fe640 1 -- 172.21.15.63:0/3009714365 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc0767a0 con 0x7f8abc077c90 2024-06-15T00:00:39.177 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.339+0000 7f8abb7fe640 1 --2- 172.21.15.63:0/3009714365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009840 tx=0x7f8aac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d0ae7edf9e1be7 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.339+0000 7f8abaffd640 1 -- 172.21.15.63:0/3009714365 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aac032440 con 0x7f8abc077c90 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.339+0000 7f8abaffd640 1 -- 172.21.15.63:0/3009714365 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8aac032a10 con 0x7f8abc077c90 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.339+0000 7f8abaffd640 1 -- 172.21.15.63:0/3009714365 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aac031a00 con 0x7f8abc077c90 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 -- 172.21.15.63:0/3009714365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 msgr2=0x7f8abc076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 --2- 172.21.15.63:0/3009714365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f8aac009840 tx=0x7f8aac0091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 -- 172.21.15.63:0/3009714365 shutdown_connections 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 --2- 172.21.15.63:0/3009714365 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc076260 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 -- 172.21.15.63:0/3009714365 >> 172.21.15.63:0/3009714365 conn(0x7f8abc0fd2d0 msgr2=0x7f8abc0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 -- 172.21.15.63:0/3009714365 shutdown_connections 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 -- 172.21.15.63:0/3009714365 wait complete. 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.340+0000 7f8ac2091640 1 Processor -- start 2024-06-15T00:00:39.178 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8ac2091640 1 -- start start 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8ac2091640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8ac2091640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8abc1130a0 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8abb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8abb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34706/0 (socket says 172.21.15.63:34706) 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8abb7fe640 1 -- 172.21.15.63:0/2607107145 learned_addr learned my addr 172.21.15.63:0/2607107145 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8abb7fe640 1 -- 172.21.15.63:0/2607107145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac009450 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.341+0000 7f8abb7fe640 1 --2- 172.21.15.63:0/2607107145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8aac031600 tx=0x7f8aac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aac03d450 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f8aac03da20 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8abc1132a0 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aac03b810 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8abc113710 con 0x7f8abc077c90 2024-06-15T00:00:39.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f8aac039070 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.342+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f8aac046620 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.343+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8abc04b480 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.345+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f8aac065790 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.405+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f8abc113ef0 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.410+0000 7f8ab8ff9640 1 -- 172.21.15.63:0/2607107145 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8aac054020 con 0x7f8abc077c90 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 msgr2=0x7f8abc112b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 --2- 172.21.15.63:0/2607107145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8aac031600 tx=0x7f8aac008040 comp rx=0 tx=0).stop 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 shutdown_connections 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 --2- 172.21.15.63:0/2607107145 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8abc077c90 0x7f8abc112b60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 >> 172.21.15.63:0/2607107145 conn(0x7f8abc0fd2d0 msgr2=0x7f8abc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 shutdown_connections 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:35.411+0000 7f8ac2091640 1 -- 172.21.15.63:0/2607107145 wait complete. 2024-06-15T00:00:39.180 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-15T00:00:39.181 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:00:39.181 INFO:teuthology.orchestra.run.smithi063.stdout:Creating mgr... 2024-06-15T00:00:39.181 INFO:teuthology.orchestra.run.smithi063.stdout:Verifying port 9283 ... 2024-06-15T00:00:39.473 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mgr.smithi063.dhizav 2024-06-15T00:00:39.474 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Failed to reset failed state of unit ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mgr.smithi063.dhizav.service: Unit ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mgr.smithi063.dhizav.service not loaded. 2024-06-15T00:00:39.824 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968.target.wants/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mgr.smithi063.dhizav.service → /etc/systemd/system/ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@.service. 2024-06-15T00:00:40.527 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:00:40.528 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-06-15T00:00:40.535 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:00:40.535 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:00:40.536 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-06-15T00:00:40.536 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to enable service . firewalld.service is not available 2024-06-15T00:00:40.545 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:00:40.545 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr to start... 2024-06-15T00:00:40.554 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr... 2024-06-15T00:00:41.327 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:41 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3077200146' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:00:41.566 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:41.566 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:00:41.566 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "25f32e20-2aaa-11ef-bc9d-c7b262605968", 2024-06-15T00:00:41.566 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:00:41.566 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "smithi063" 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:41.567 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:00:41.568 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:00:41.576 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:00:41.576 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:00:41.576 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:00:41.577 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:00:41.578 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:00:30.303538+0000", 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:41.579 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f5e20d640 1 Processor -- start 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f5e20d640 1 -- start start 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f5e20d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f5e20d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f581024f0 con 0x7f1f58075c00 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f577fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f577fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50456/0 (socket says 172.21.15.63:50456) 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.970+0000 7f1f577fe640 1 -- 172.21.15.63:0/3072463611 learned_addr learned my addr 172.21.15.63:0/3072463611 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.971+0000 7f1f577fe640 1 -- 172.21.15.63:0/3072463611 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f58107040 con 0x7f1f58075c00 2024-06-15T00:00:41.580 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.971+0000 7f1f577fe640 1 --2- 172.21.15.63:0/3072463611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009840 tx=0x7f1f480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8254c9ba8fbf10a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f567fc640 1 -- 172.21.15.63:0/3072463611 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f48032440 con 0x7f1f58075c00 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f567fc640 1 -- 172.21.15.63:0/3072463611 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1f48032a10 con 0x7f1f58075c00 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f567fc640 1 -- 172.21.15.63:0/3072463611 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f48031a00 con 0x7f1f58075c00 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3072463611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 msgr2=0x7f1f58106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 --2- 172.21.15.63:0/3072463611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1f48009840 tx=0x7f1f480091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3072463611 shutdown_connections 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 --2- 172.21.15.63:0/3072463611 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58106b00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3072463611 >> 172.21.15.63:0/3072463611 conn(0x7f1f580fc150 msgr2=0x7f1f580fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3072463611 shutdown_connections 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.972+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3072463611 wait complete. 2024-06-15T00:00:41.581 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f5e20d640 1 Processor -- start 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f5e20d640 1 -- start start 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f5e20d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f5e20d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f58112fd0 con 0x7f1f58075c00 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f577fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f577fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50470/0 (socket says 172.21.15.63:50470) 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.973+0000 7f1f577fe640 1 -- 172.21.15.63:0/3077200146 learned_addr learned my addr 172.21.15.63:0/3077200146 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f577fe640 1 -- 172.21.15.63:0/3077200146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f48009450 con 0x7f1f58075c00 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f577fe640 1 --2- 172.21.15.63:0/3077200146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1f480098a0 tx=0x7f1f48008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4803d450 con 0x7f1f58075c00 2024-06-15T00:00:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1f4803da20 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f581131d0 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1f4803b860 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f58113640 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1f48039070 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.974+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1f480466c0 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.975+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f5804b480 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:40.977+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f1f48065790 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.041+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1f58113e20 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.041+0000 7f1f3ffff640 1 -- 172.21.15.63:0/3077200146 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f1f48054020 con 0x7f1f58075c00 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.042+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 msgr2=0x7f1f58112a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.042+0000 7f1f5e20d640 1 --2- 172.21.15.63:0/3077200146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1f480098a0 tx=0x7f1f48008040 comp rx=0 tx=0).stop 2024-06-15T00:00:41.583 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.042+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 shutdown_connections 2024-06-15T00:00:41.584 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.042+0000 7f1f5e20d640 1 --2- 172.21.15.63:0/3077200146 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1f58075c00 0x7f1f58112a90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:41.584 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.042+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 >> 172.21.15.63:0/3077200146 conn(0x7f1f580fc150 msgr2=0x7f1f580fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:41.584 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.043+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 shutdown_connections 2024-06-15T00:00:41.584 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:41.043+0000 7f1f5e20d640 1 -- 172.21.15.63:0/3077200146 wait complete. 2024-06-15T00:00:41.584 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (1/15)... 2024-06-15T00:00:44.367 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:44 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2836851165' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:00:44.615 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "25f32e20-2aaa-11ef-bc9d-c7b262605968", 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:00:44.616 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "smithi063" 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:00:44.617 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.618 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:44.619 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:00:30.303538+0000", 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d8b018640 1 Processor -- start 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d8b018640 1 -- start start 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d8b018640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d8b018640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84070370 con 0x7f5d84103130 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d88db4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:44.620 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d88db4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50474/0 (socket says 172.21.15.63:50474) 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.966+0000 7f5d88db4640 1 -- 172.21.15.63:0/770610137 learned_addr learned my addr 172.21.15.63:0/770610137 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.967+0000 7f5d88db4640 1 -- 172.21.15.63:0/770610137 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d84103a80 con 0x7f5d84103130 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.967+0000 7f5d88db4640 1 --2- 172.21.15.63:0/770610137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009840 tx=0x7f5d780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bf31e9ffdb40 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d837fe640 1 -- 172.21.15.63:0/770610137 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d78032440 con 0x7f5d84103130 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d837fe640 1 -- 172.21.15.63:0/770610137 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5d78032a10 con 0x7f5d84103130 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d837fe640 1 -- 172.21.15.63:0/770610137 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d78031a00 con 0x7f5d84103130 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d8b018640 1 -- 172.21.15.63:0/770610137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 msgr2=0x7f5d84103540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d8b018640 1 --2- 172.21.15.63:0/770610137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5d78009840 tx=0x7f5d780091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d8b018640 1 -- 172.21.15.63:0/770610137 shutdown_connections 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d8b018640 1 --2- 172.21.15.63:0/770610137 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84103540 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.968+0000 7f5d8b018640 1 -- 172.21.15.63:0/770610137 >> 172.21.15.63:0/770610137 conn(0x7f5d840fb8b0 msgr2=0x7f5d840fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.969+0000 7f5d8b018640 1 -- 172.21.15.63:0/770610137 shutdown_connections 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.969+0000 7f5d8b018640 1 -- 172.21.15.63:0/770610137 wait complete. 2024-06-15T00:00:44.621 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.969+0000 7f5d8b018640 1 Processor -- start 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.969+0000 7f5d8b018640 1 -- start start 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d8b018640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d8b018640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d84102d80 con 0x7f5d84103130 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d88db4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d88db4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50480/0 (socket says 172.21.15.63:50480) 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d88db4640 1 -- 172.21.15.63:0/2836851165 learned_addr learned my addr 172.21.15.63:0/2836851165 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d88db4640 1 -- 172.21.15.63:0/2836851165 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d78009450 con 0x7f5d84103130 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.970+0000 7f5d88db4640 1 --2- 172.21.15.63:0/2836851165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d780098a0 tx=0x7f5d78008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7803d450 con 0x7f5d84103130 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5d7803da20 con 0x7f5d84103130 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d84100e10 con 0x7f5d84103130 2024-06-15T00:00:44.622 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7803b860 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d84101280 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f5d78039070 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.971+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5d780466c0 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.972+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8404b480 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:43.974+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f5d78065790 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.038+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5d84100540 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.039+0000 7f5d817fa640 1 -- 172.21.15.63:0/2836851165 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f5d78054020 con 0x7f5d84103130 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 msgr2=0x7f5d84102840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 --2- 172.21.15.63:0/2836851165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d780098a0 tx=0x7f5d78008040 comp rx=0 tx=0).stop 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 shutdown_connections 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 --2- 172.21.15.63:0/2836851165 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5d84103130 0x7f5d84102840 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 >> 172.21.15.63:0/2836851165 conn(0x7f5d840fb8b0 msgr2=0x7f5d840fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:44.623 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 shutdown_connections 2024-06-15T00:00:44.624 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:44.040+0000 7f5d8b018640 1 -- 172.21.15.63:0/2836851165 wait complete. 2024-06-15T00:00:44.624 INFO:teuthology.orchestra.run.smithi063.stdout:mgr not available, waiting (2/15)... 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: Activating manager daemon smithi063.dhizav 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: mgrmap e2: smithi063.dhizav(active, starting, since 0.00289401s) 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr metadata", "who": "smithi063.dhizav", "id": "smithi063.dhizav"}]: dispatch 2024-06-15T00:00:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: Manager daemon smithi063.dhizav is now available 2024-06-15T00:00:45.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:00:45.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/trash_purge_schedule"}]: dispatch 2024-06-15T00:00:45.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' 2024-06-15T00:00:45.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' 2024-06-15T00:00:45.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:45 smithi063 ceph-mon[28882]: from='mgr.14100 172.21.15.63:0/1866791705' entity='mgr.smithi063.dhizav' 2024-06-15T00:00:47.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:47 smithi063 ceph-mon[28882]: mgrmap e3: smithi063.dhizav(active, since 1.00701s) 2024-06-15T00:00:48.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:48 smithi063 ceph-mon[28882]: mgrmap e4: smithi063.dhizav(active, since 2s) 2024-06-15T00:00:50.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:50 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/330346170' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:00:52.173 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:52.173 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:00:52.173 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsid": "25f32e20-2aaa-11ef-bc9d-c7b262605968", 2024-06-15T00:00:52.173 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:00:52.173 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "smithi063" 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "quorum_age": 15, 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:00:52.174 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:00:52.175 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:00:52.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:00:52.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:00:30.303538+0000", 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:00:52.186 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e8b71640 1 Processor -- start 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e8b71640 1 -- start start 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e8b71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e8b71640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4102440 con 0x7f16e4101af0 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39364/0 (socket says 172.21.15.63:39364) 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.104+0000 7f16e259c640 1 -- 172.21.15.63:0/3758120802 learned_addr learned my addr 172.21.15.63:0/3758120802 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.105+0000 7f16e259c640 1 -- 172.21.15.63:0/3758120802 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e4102580 con 0x7f16e4101af0 2024-06-15T00:00:52.187 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.105+0000 7f16e259c640 1 --2- 172.21.15.63:0/3758120802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f16d4009840 tx=0x7f16d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394682c6a34e6093 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.105+0000 7f16e159a640 1 -- 172.21.15.63:0/3758120802 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d4032440 con 0x7f16e4101af0 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e159a640 1 -- 172.21.15.63:0/3758120802 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f16d4032a10 con 0x7f16e4101af0 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e159a640 1 -- 172.21.15.63:0/3758120802 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d4031a00 con 0x7f16e4101af0 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 -- 172.21.15.63:0/3758120802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 msgr2=0x7f16e4101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 --2- 172.21.15.63:0/3758120802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f16d4009840 tx=0x7f16d40091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 -- 172.21.15.63:0/3758120802 shutdown_connections 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 --2- 172.21.15.63:0/3758120802 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4101f00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 -- 172.21.15.63:0/3758120802 >> 172.21.15.63:0/3758120802 conn(0x7f16e40fd330 msgr2=0x7f16e40ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 -- 172.21.15.63:0/3758120802 shutdown_connections 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.106+0000 7f16e8b71640 1 -- 172.21.15.63:0/3758120802 wait complete. 2024-06-15T00:00:52.188 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e8b71640 1 Processor -- start 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e8b71640 1 -- start start 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e8b71640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e8b71640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16e4110dc0 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39372/0 (socket says 172.21.15.63:39372) 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e259c640 1 -- 172.21.15.63:0/330346170 learned_addr learned my addr 172.21.15.63:0/330346170 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.107+0000 7f16e259c640 1 -- 172.21.15.63:0/330346170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16d4009450 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16e259c640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f16d40313e0 tx=0x7f16d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d403d450 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f16d403da20 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16e4110fc0 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f16d403b8d0 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.108+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16e4111430 con 0x7f16e4101af0 2024-06-15T00:00:52.189 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f16d4039070 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16d2ffd640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 0x7f16cc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f16d4067130 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16e1d9b640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 0x7f16cc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16e405ead0 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.109+0000 7f16e1d9b640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 0x7f16cc036cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16d8006fd0 tx=0x7f16d8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.114+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f16d4041020 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.352+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f16e404b480 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.352+0000 7f16d2ffd640 1 -- 172.21.15.63:0/330346170 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f16d4037070 con 0x7f16e4101af0 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 msgr2=0x7f16cc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 0x7f16cc036cc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f16d8006fd0 tx=0x7f16d8006db0 comp rx=0 tx=0).stop 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 msgr2=0x7f16e4110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:52.190 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f16d40313e0 tx=0x7f16d4008040 comp rx=0 tx=0).stop 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 shutdown_connections 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f16cc034810 0x7f16cc036cc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 --2- 172.21.15.63:0/330346170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f16e4101af0 0x7f16e4110880 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 >> 172.21.15.63:0/330346170 conn(0x7f16e40fd330 msgr2=0x7f16e40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 shutdown_connections 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:50.354+0000 7f16e8b71640 1 -- 172.21.15.63:0/330346170 wait complete. 2024-06-15T00:00:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:mgr is available 2024-06-15T00:00:53.057 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:52 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2094112706' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [global] 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout fsid = 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-15T00:00:53.316 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout [osd] 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc5a4eac640 1 Processor -- start 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc5a4eac640 1 -- start start 2024-06-15T00:00:53.317 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc5a4eac640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc5a4eac640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0102440 con 0x7fc5a0101af0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.565+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39388/0 (socket says 172.21.15.63:39388) 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.566+0000 7fc59e59c640 1 -- 172.21.15.63:0/172962315 learned_addr learned my addr 172.21.15.63:0/172962315 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.566+0000 7fc59e59c640 1 -- 172.21.15.63:0/172962315 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0102580 con 0x7fc5a0101af0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.566+0000 7fc59e59c640 1 --2- 172.21.15.63:0/172962315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc590009840 tx=0x7fc5900091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e676465324563de server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc59d59a640 1 -- 172.21.15.63:0/172962315 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590032440 con 0x7fc5a0101af0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc59d59a640 1 -- 172.21.15.63:0/172962315 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc590032a10 con 0x7fc5a0101af0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc59d59a640 1 -- 172.21.15.63:0/172962315 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc590031a00 con 0x7fc5a0101af0 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 -- 172.21.15.63:0/172962315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 msgr2=0x7fc5a0101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/172962315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc590009840 tx=0x7fc5900091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:53.318 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 -- 172.21.15.63:0/172962315 shutdown_connections 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/172962315 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0101f00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 -- 172.21.15.63:0/172962315 >> 172.21.15.63:0/172962315 conn(0x7fc5a00fd330 msgr2=0x7fc5a00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.567+0000 7fc5a4eac640 1 -- 172.21.15.63:0/172962315 shutdown_connections 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc5a4eac640 1 -- 172.21.15.63:0/172962315 wait complete. 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc5a4eac640 1 Processor -- start 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc5a4eac640 1 -- start start 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc5a4eac640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc5a4eac640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5a0110dc0 con 0x7fc5a0101af0 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc59e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39392/0 (socket says 172.21.15.63:39392) 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.568+0000 7fc59e59c640 1 -- 172.21.15.63:0/2094112706 learned_addr learned my addr 172.21.15.63:0/2094112706 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc59e59c640 1 -- 172.21.15.63:0/2094112706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc590009450 con 0x7fc5a0101af0 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc59e59c640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc5900098a0 tx=0x7fc590008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:53.319 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59003d450 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fc59003da20 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5a0110fc0 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc59003b860 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.569+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5a0111430 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.570+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7fc590039070 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.570+0000 7fc58effd640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 0x7fc588036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.570+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc590067100 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.570+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5a005ead0 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.570+0000 7fc59dd9b640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 0x7fc588036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.571+0000 7fc59dd9b640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 0x7fc588036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc594006fd0 tx=0x7fc594006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.575+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc590041020 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.750+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc5a0103850 con 0x7fc5a0101af0 2024-06-15T00:00:53.320 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.750+0000 7fc58effd640 1 -- 172.21.15.63:0/2094112706 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fc590037070 con 0x7fc5a0101af0 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 msgr2=0x7fc588036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 0x7fc588036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc594006fd0 tx=0x7fc594006db0 comp rx=0 tx=0).stop 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 msgr2=0x7fc5a0110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc5900098a0 tx=0x7fc590008040 comp rx=0 tx=0).stop 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 shutdown_connections 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7fc5880347c0 0x7fc588036c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 --2- 172.21.15.63:0/2094112706 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc5a0101af0 0x7fc5a0110880 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 >> 172.21.15.63:0/2094112706 conn(0x7fc5a00fd330 msgr2=0x7fc5a00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 shutdown_connections 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:52.753+0000 7fc5a4eac640 1 -- 172.21.15.63:0/2094112706 wait complete. 2024-06-15T00:00:53.321 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling cephadm module... 2024-06-15T00:00:54.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:53 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/853748103' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-15T00:00:54.605 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.674+0000 7f1ac862a640 1 Processor -- start 2024-06-15T00:00:54.605 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.674+0000 7f1ac862a640 1 -- start start 2024-06-15T00:00:54.605 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac862a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:54.605 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac862a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00780f0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.605 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac63c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac63c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39404/0 (socket says 172.21.15.63:39404) 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac63c6640 1 -- 172.21.15.63:0/268190768 learned_addr learned my addr 172.21.15.63:0/268190768 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.675+0000 7f1ac63c6640 1 -- 172.21.15.63:0/268190768 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00767a0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.676+0000 7f1ac63c6640 1 --2- 172.21.15.63:0/268190768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009840 tx=0x7f1abc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=251e3f483520cc2f server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.676+0000 7f1ac53c4640 1 -- 172.21.15.63:0/268190768 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc032440 con 0x7f1ac0077ce0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.676+0000 7f1ac53c4640 1 -- 172.21.15.63:0/268190768 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1abc032a10 con 0x7f1ac0077ce0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.676+0000 7f1ac53c4640 1 -- 172.21.15.63:0/268190768 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc031a00 con 0x7f1ac0077ce0 2024-06-15T00:00:54.606 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 -- 172.21.15.63:0/268190768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 msgr2=0x7f1ac0076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 --2- 172.21.15.63:0/268190768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1abc009840 tx=0x7f1abc0091d0 comp rx=0 tx=0).stop 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 -- 172.21.15.63:0/268190768 shutdown_connections 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 --2- 172.21.15.63:0/268190768 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0076260 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 -- 172.21.15.63:0/268190768 >> 172.21.15.63:0/268190768 conn(0x7f1ac00fd2e0 msgr2=0x7f1ac00ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 -- 172.21.15.63:0/268190768 shutdown_connections 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 -- 172.21.15.63:0/268190768 wait complete. 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.677+0000 7f1ac862a640 1 Processor -- start 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac862a640 1 -- start start 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac862a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac862a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0112f40 con 0x7f1ac0077ce0 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac63c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac63c6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39412/0 (socket says 172.21.15.63:39412) 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac63c6640 1 -- 172.21.15.63:0/853748103 learned_addr learned my addr 172.21.15.63:0/853748103 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac63c6640 1 -- 172.21.15.63:0/853748103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1abc009450 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac63c6640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1abc006f00 tx=0x7f1abc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc031e20 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1abc03d8c0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1abc03b840 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.678+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac0113140 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.679+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac01135b0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.679+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f1abc039070 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.680+0000 7f1ab6ffd640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 0x7f1aac036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.680+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1abc0671f0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.680+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac005ead0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.680+0000 7f1ac5bc5640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 0x7f1aac036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.680+0000 7f1ac5bc5640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 0x7f1aac036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:54.608 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.684+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1abc041020 con 0x7f1ac0077ce0 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.902+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f1ac004b480 con 0x7f1ac0077ce0 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.955+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1abc045c90 con 0x7f1ac0077ce0 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.956+0000 7f1ab6ffd640 1 -- 172.21.15.63:0/853748103 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f1abc04ddc0 con 0x7f1ac0077ce0 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 msgr2=0x7f1aac036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 0x7f1aac036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0006fd0 tx=0x7f1ab0006db0 comp rx=0 tx=0).stop 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 msgr2=0x7f1ac0112a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1abc006f00 tx=0x7f1abc008040 comp rx=0 tx=0).stop 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 shutdown_connections 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f1aac034810 0x7f1aac036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 --2- 172.21.15.63:0/853748103 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1ac0077ce0 0x7f1ac0112a00 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.958+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 >> 172.21.15.63:0/853748103 conn(0x7f1ac00fd2e0 msgr2=0x7f1ac00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.959+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 shutdown_connections 2024-06-15T00:00:54.609 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:53.959+0000 7f1ac862a640 1 -- 172.21.15.63:0/853748103 wait complete. 2024-06-15T00:00:55.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:54 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/853748103' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-15T00:00:55.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:54 smithi063 ceph-mon[28882]: mgrmap e5: smithi063.dhizav(active, since 8s) 2024-06-15T00:00:55.823 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:00:55.823 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-06-15T00:00:55.823 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "smithi063.dhizav", 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d30cb640 1 Processor -- start 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d30cb640 1 -- start start 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d30cb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d30cb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc1012e0 con 0x7f07cc100990 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d0e67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d0e67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39436/0 (socket says 172.21.15.63:39436) 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d0e67640 1 -- 172.21.15.63:0/706609773 learned_addr learned my addr 172.21.15.63:0/706609773 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.996+0000 7f07d0e67640 1 -- 172.21.15.63:0/706609773 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc101420 con 0x7f07cc100990 2024-06-15T00:00:55.824 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.997+0000 7f07d0e67640 1 --2- 172.21.15.63:0/706609773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f07c000baf0 tx=0x7f07c000b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37ec60a49fc8d9b6 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.997+0000 7f07cb7fe640 1 -- 172.21.15.63:0/706609773 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c00327a0 con 0x7f07cc100990 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.997+0000 7f07cb7fe640 1 -- 172.21.15.63:0/706609773 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f07c0032d70 con 0x7f07cc100990 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.997+0000 7f07cb7fe640 1 -- 172.21.15.63:0/706609773 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c0031dc0 con 0x7f07cc100990 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 -- 172.21.15.63:0/706609773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 msgr2=0x7f07cc100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 --2- 172.21.15.63:0/706609773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f07c000baf0 tx=0x7f07c000b460 comp rx=0 tx=0).stop 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 -- 172.21.15.63:0/706609773 shutdown_connections 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 --2- 172.21.15.63:0/706609773 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc100da0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 -- 172.21.15.63:0/706609773 >> 172.21.15.63:0/706609773 conn(0x7f07cc0fc190 msgr2=0x7f07cc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 -- 172.21.15.63:0/706609773 shutdown_connections 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.998+0000 7f07d30cb640 1 -- 172.21.15.63:0/706609773 wait complete. 2024-06-15T00:00:55.825 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d30cb640 1 Processor -- start 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d30cb640 1 -- start start 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d30cb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d30cb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc10ec20 con 0x7f07cc100990 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d0e67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d0e67640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39450/0 (socket says 172.21.15.63:39450) 2024-06-15T00:00:55.826 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d0e67640 1 -- 172.21.15.63:0/3206293120 learned_addr learned my addr 172.21.15.63:0/3206293120 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:54.999+0000 7f07d0e67640 1 -- 172.21.15.63:0/3206293120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c000b700 con 0x7f07cc100990 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07d0e67640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003080 tx=0x7f07c003d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c0032b80 con 0x7f07cc100990 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc10ee20 con 0x7f07cc100990 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f07c003b890 con 0x7f07cc100990 2024-06-15T00:00:55.827 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f07c0045700 con 0x7f07cc100990 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.000+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc10f290 con 0x7f07cc100990 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f07c0045980 con 0x7f07cc100990 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07c97fa640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 0x7f07bc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f07c0069fd0 con 0x7f07cc100990 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc10f290 con 0x7f07cc100990 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07cbfff640 1 -- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 msgr2=0x7f07bc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:00:55.828 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.001+0000 7f07cbfff640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 0x7f07bc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.006+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f07cc10f290 con 0x7f07cc100990 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.202+0000 7f07cbfff640 1 -- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 msgr2=0x7f07bc036cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.202+0000 7f07cbfff640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 0x7f07bc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.226+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f07cc102740 con 0x7f07cc100990 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.226+0000 7f07c97fa640 1 -- 172.21.15.63:0/3206293120 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f07c0036070 con 0x7f07cc100990 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.228+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 msgr2=0x7f07bc036cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:00:55.829 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.228+0000 7f07d30cb640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 0x7f07bc036cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.228+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 msgr2=0x7f07cc10e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.228+0000 7f07d30cb640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003080 tx=0x7f07c003d040 comp rx=0 tx=0).stop 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.228+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 shutdown_connections 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.229+0000 7f07d30cb640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f07bc034810 0x7f07bc036cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.229+0000 7f07d30cb640 1 --2- 172.21.15.63:0/3206293120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07cc100990 0x7f07cc10e6e0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:00:55.830 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.229+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 >> 172.21.15.63:0/3206293120 conn(0x7f07cc0fc190 msgr2=0x7f07cc104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:00:55.831 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.229+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 shutdown_connections 2024-06-15T00:00:55.831 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:55.229+0000 7f07d30cb640 1 -- 172.21.15.63:0/3206293120 wait complete. 2024-06-15T00:00:55.831 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-06-15T00:00:55.831 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 5... 2024-06-15T00:00:56.050 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:55 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3206293120' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-15T00:00:58.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: Active manager daemon smithi063.dhizav restarted 2024-06-15T00:00:58.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: Activating manager daemon smithi063.dhizav 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: osdmap e2: 0 total, 0 up, 0 in 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: mgrmap e6: smithi063.dhizav(active, starting, since 0.00318971s) 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr metadata", "who": "smithi063.dhizav", "id": "smithi063.dhizav"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: Manager daemon smithi063.dhizav is now available 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:00:58.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:00:58.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:00:58.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:00:58.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:58 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/trash_purge_schedule"}]: dispatch 2024-06-15T00:00:59.926 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:59 smithi063 ceph-mon[28882]: Found migration_current of "None". Setting to last migration. 2024-06-15T00:00:59.927 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:00:59 smithi063 ceph-mon[28882]: mgrmap e7: smithi063.dhizav(active, since 1.0053s) 2024-06-15T00:01:00.179 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.227+0000 7f00603da640 1 Processor -- start 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.227+0000 7f00603da640 1 -- start start 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.227+0000 7f00603da640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.227+0000 7f00603da640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0058102890 con 0x7f0058101f40 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.228+0000 7f005e176640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.228+0000 7f005e176640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39460/0 (socket says 172.21.15.63:39460) 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.228+0000 7f005e176640 1 -- 172.21.15.63:0/2058324438 learned_addr learned my addr 172.21.15.63:0/2058324438 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:00.180 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.228+0000 7f005e176640 1 -- 172.21.15.63:0/2058324438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00581029d0 con 0x7f0058101f40 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f005e176640 1 --2- 172.21.15.63:0/2058324438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f005400baf0 tx=0x7f005400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cf8e1d01bbac53 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f005d174640 1 -- 172.21.15.63:0/2058324438 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00540327a0 con 0x7f0058101f40 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f005d174640 1 -- 172.21.15.63:0/2058324438 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0054032d70 con 0x7f0058101f40 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f005d174640 1 -- 172.21.15.63:0/2058324438 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0054031dc0 con 0x7f0058101f40 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f00603da640 1 -- 172.21.15.63:0/2058324438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 msgr2=0x7f0058102350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.229+0000 7f00603da640 1 --2- 172.21.15.63:0/2058324438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f005400baf0 tx=0x7f005400b460 comp rx=0 tx=0).stop 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- 172.21.15.63:0/2058324438 shutdown_connections 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 --2- 172.21.15.63:0/2058324438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f0058102350 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- 172.21.15.63:0/2058324438 >> 172.21.15.63:0/2058324438 conn(0x7f00580fb8c0 msgr2=0x7f00580fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- 172.21.15.63:0/2058324438 shutdown_connections 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- 172.21.15.63:0/2058324438 wait complete. 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 Processor -- start 2024-06-15T00:01:00.181 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- start start 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f00603da640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005810ec00 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.230+0000 7f005e176640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f005e176640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39476/0 (socket says 172.21.15.63:39476) 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f005e176640 1 -- 172.21.15.63:0/3147526661 learned_addr learned my addr 172.21.15.63:0/3147526661 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f005e176640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005400b700 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f005e176640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0054003080 tx=0x7f005403d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0054032ab0 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f005403b890 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005810ee00 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0054045740 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.231+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005810f270 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f005403ba00 con 0x7f0058101f40 2024-06-15T00:01:00.182 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f004effd640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f005805ead0 con 0x7f0044034860 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0054069f50 con 0x7f0058101f40 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f005d975640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 msgr2=0x7f0044036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.232+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.433+0000 7f005d975640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 msgr2=0x7f0044036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.433+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.834+0000 7f005d975640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 msgr2=0x7f0044036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:56.834+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:57.635+0000 7f005d975640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 msgr2=0x7f0044036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/1148153327 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:57.635+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:58.666+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 6) v1 ==== 41713+0+0 (secure 0 0 0) 0x7f005404dc60 con 0x7f0058101f40 2024-06-15T00:01:00.183 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:58.667+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 msgr2=0x7f0044036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:58.667+0000 7f004effd640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.667+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7f005404e400 con 0x7f0058101f40 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.668+0000 7f004effd640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 0x7f004403a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.668+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f005805ead0 con 0x7f0044037fd0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.668+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 0x7f004403a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.669+0000 7f005d975640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 0x7f004403a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0048003940 tx=0x7f0048009350 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.670+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f005805ead0 con 0x7f0044037fd0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.675+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f005810f330 con 0x7f0044037fd0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.675+0000 7f004effd640 1 -- 172.21.15.63:0/3147526661 <== mgr.14118 v2:172.21.15.63:6800/55223302 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f005810f330 con 0x7f0044037fd0 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 msgr2=0x7f004403a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 0x7f004403a3b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f0048003940 tx=0x7f0048009350 comp rx=0 tx=0).stop 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 msgr2=0x7f005810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0054003080 tx=0x7f005403d040 comp rx=0 tx=0).stop 2024-06-15T00:01:00.184 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 shutdown_connections 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0044037fd0 0x7f004403a3b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:6800/1148153327,v1:172.21.15.63:6801/1148153327] conn(0x7f0044034860 0x7f0044036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 --2- 172.21.15.63:0/3147526661 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0058101f40 0x7f005810e6c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 >> 172.21.15.63:0/3147526661 conn(0x7f00580fb8c0 msgr2=0x7f0058104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 shutdown_connections 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:00:59.676+0000 7f00603da640 1 -- 172.21.15.63:0/3147526661 wait complete. 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 5 is available 2024-06-15T00:01:00.185 INFO:teuthology.orchestra.run.smithi063.stdout:Setting orchestrator backend to cephadm... 2024-06-15T00:01:00.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:00 smithi063 ceph-mon[28882]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-15T00:01:00.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:00 smithi063 ceph-mon[28882]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-15T00:01:01.308 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.558+0000 7f59b79d2640 1 Processor -- start 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.558+0000 7f59b79d2640 1 -- start start 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b79d2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b79d2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0104bc0 con 0x7f59b0075c80 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b576e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b576e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47796/0 (socket says 172.21.15.63:47796) 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b576e640 1 -- 172.21.15.63:0/51088843 learned_addr learned my addr 172.21.15.63:0/51088843 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.559+0000 7f59b576e640 1 -- 172.21.15.63:0/51088843 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0104d00 con 0x7f59b0075c80 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.560+0000 7f59b576e640 1 --2- 172.21.15.63:0/51088843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009840 tx=0x7f59ac0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bdbd0de297d975 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:01.309 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.560+0000 7f59a7fff640 1 -- 172.21.15.63:0/51088843 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac032440 con 0x7f59b0075c80 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.560+0000 7f59a7fff640 1 -- 172.21.15.63:0/51088843 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f59ac032a10 con 0x7f59b0075c80 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.560+0000 7f59a7fff640 1 -- 172.21.15.63:0/51088843 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac031a00 con 0x7f59b0075c80 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- 172.21.15.63:0/51088843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 msgr2=0x7f59b0104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 --2- 172.21.15.63:0/51088843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f59ac009840 tx=0x7f59ac0091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- 172.21.15.63:0/51088843 shutdown_connections 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 --2- 172.21.15.63:0/51088843 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b0104680 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- 172.21.15.63:0/51088843 >> 172.21.15.63:0/51088843 conn(0x7f59b00fc150 msgr2=0x7f59b00fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- 172.21.15.63:0/51088843 shutdown_connections 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- 172.21.15.63:0/51088843 wait complete. 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 Processor -- start 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.561+0000 7f59b79d2640 1 -- start start 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b79d2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:01.310 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b79d2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59b0110de0 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b576e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b576e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47806/0 (socket says 172.21.15.63:47806) 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b576e640 1 -- 172.21.15.63:0/1975644473 learned_addr learned my addr 172.21.15.63:0/1975644473 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b576e640 1 -- 172.21.15.63:0/1975644473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac009450 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b576e640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0098a0 tx=0x7f59ac008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac03d450 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f59ac03da20 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b0111040 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.562+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59ac03b860 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.563+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b0111b70 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7f59ac039070 con 0x7f59b0075c80 2024-06-15T00:01:01.311 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59a5ffb640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 0x7f599c036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b005ead0 con 0x7f59b0075c80 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f59ac067530 con 0x7f59b0075c80 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59b4f6d640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 0x7f599c036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.564+0000 7f59b4f6d640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 0x7f599c036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59a0006fd0 tx=0x7f59a0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.569+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f59ac04e710 con 0x7f59b0075c80 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.722+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f59ac03cd00 con 0x7f59b0075c80 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.757+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f59b00fd550 con 0x7f599c034730 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.764+0000 7f59a5ffb640 1 -- 172.21.15.63:0/1975644473 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f59b00fd550 con 0x7f599c034730 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.766+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 msgr2=0x7f599c036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.766+0000 7f59b79d2640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 0x7f599c036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f59a0006fd0 tx=0x7f59a0006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.766+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 msgr2=0x7f59b01108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.766+0000 7f59b79d2640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0098a0 tx=0x7f59ac008040 comp rx=0 tx=0).stop 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.766+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 shutdown_connections 2024-06-15T00:01:01.312 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.767+0000 7f59b79d2640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f599c034730 0x7f599c036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:01.313 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.767+0000 7f59b79d2640 1 --2- 172.21.15.63:0/1975644473 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f59b0075c80 0x7f59b01108a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:01.313 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.767+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 >> 172.21.15.63:0/1975644473 conn(0x7f59b00fc150 msgr2=0x7f59b00fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:01.313 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.767+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 shutdown_connections 2024-06-15T00:01:01.313 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:00.767+0000 7f59b79d2640 1 -- 172.21.15.63:0/1975644473 wait complete. 2024-06-15T00:01:01.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:01 smithi063 ceph-mon[28882]: mgrmap e8: smithi063.dhizav(active, since 2s) 2024-06-15T00:01:01.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:01 smithi063 ceph-mon[28882]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:01.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:01 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:01.993 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:01 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:01:02.473 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b1b6ba640 1 Processor -- start 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b1b6ba640 1 -- start start 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b1b6ba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b1b6ba640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14102470 con 0x7f0b14075c70 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b19456640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b19456640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47814/0 (socket says 172.21.15.63:47814) 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.690+0000 7f0b19456640 1 -- 172.21.15.63:0/3484812026 learned_addr learned my addr 172.21.15.63:0/3484812026 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.691+0000 7f0b19456640 1 -- 172.21.15.63:0/3484812026 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b14106fc0 con 0x7f0b14075c70 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.691+0000 7f0b19456640 1 --2- 172.21.15.63:0/3484812026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009840 tx=0x7f0b100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f602f93ca28ebce server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b0bfff640 1 -- 172.21.15.63:0/3484812026 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b10032440 con 0x7f0b14075c70 2024-06-15T00:01:02.474 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b0bfff640 1 -- 172.21.15.63:0/3484812026 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0b10032a10 con 0x7f0b14075c70 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b0bfff640 1 -- 172.21.15.63:0/3484812026 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b10031a00 con 0x7f0b14075c70 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3484812026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 msgr2=0x7f0b14106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3484812026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0b10009840 tx=0x7f0b100091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3484812026 shutdown_connections 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3484812026 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14106a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3484812026 >> 172.21.15.63:0/3484812026 conn(0x7f0b140fc130 msgr2=0x7f0b140fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3484812026 shutdown_connections 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.692+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3484812026 wait complete. 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b1b6ba640 1 Processor -- start 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b1b6ba640 1 -- start start 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b1b6ba640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:02.475 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b1b6ba640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b14112fa0 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b19456640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b19456640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47820/0 (socket says 172.21.15.63:47820) 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b19456640 1 -- 172.21.15.63:0/3543377942 learned_addr learned my addr 172.21.15.63:0/3543377942 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.693+0000 7f0b19456640 1 -- 172.21.15.63:0/3543377942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b10009450 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b19456640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b100098a0 tx=0x7f0b10008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1003d450 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0b1003da20 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b141131a0 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b1003b860 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.694+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b14113610 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.695+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f0b10039070 con 0x7f0b14075c70 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.695+0000 7f0b09ffb640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 0x7f0b00036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:02.476 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.695+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f0b10067410 con 0x7f0b14075c70 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.695+0000 7f0b18c55640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 0x7f0b00036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.695+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1405ead0 con 0x7f0b14075c70 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.696+0000 7f0b18c55640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 0x7f0b00036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006fd0 tx=0x7f0b04006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.700+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0b10041070 con 0x7f0b14075c70 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.875+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b140fd4f0 con 0x7f0b00034860 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.880+0000 7f0b09ffb640 1 -- 172.21.15.63:0/3543377942 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0b140fd4f0 con 0x7f0b00034860 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 msgr2=0x7f0b00036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 0x7f0b00036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0b04006fd0 tx=0x7f0b04006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 msgr2=0x7f0b14112a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0b100098a0 tx=0x7f0b10008040 comp rx=0 tx=0).stop 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 shutdown_connections 2024-06-15T00:01:02.477 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0b00034860 0x7f0b00036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:02.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 --2- 172.21.15.63:0/3543377942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b14075c70 0x7f0b14112a60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:02.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 >> 172.21.15.63:0/3543377942 conn(0x7f0b140fc130 msgr2=0x7f0b140fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:02.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 shutdown_connections 2024-06-15T00:01:02.478 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:01.882+0000 7f0b1b6ba640 1 -- 172.21.15.63:0/3543377942 wait complete. 2024-06-15T00:01:02.478 INFO:teuthology.orchestra.run.smithi063.stdout:Generating ssh key... 2024-06-15T00:01:03.023 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:01] ENGINE Bus STARTING 2024-06-15T00:01:03.023 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:03.023 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:01] ENGINE Serving on https://172.21.15.63:7150 2024-06-15T00:01:03.023 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:01] ENGINE Bus STARTED 2024-06-15T00:01:03.023 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:01] ENGINE Error in HTTPServer.serve 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Traceback (most recent call last): 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._run(expiration_interval) 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s = self.context.wrap_socket( 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:01:03.024 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: return self.sslsocket_class._create( 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self.do_handshake() 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._sslobj.do_handshake() 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:03.025 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:02 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:01:03.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f4546082640 1 Processor -- start 2024-06-15T00:01:03.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f4546082640 1 -- start start 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f4546082640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f4546082640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45401007f0 con 0x7f45400ffea0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f453f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f453f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47824/0 (socket says 172.21.15.63:47824) 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f453f7fe640 1 -- 172.21.15.63:0/1373768454 learned_addr learned my addr 172.21.15.63:0/1373768454 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.840+0000 7f453f7fe640 1 -- 172.21.15.63:0/1373768454 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4540100930 con 0x7f45400ffea0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.841+0000 7f453f7fe640 1 --2- 172.21.15.63:0/1373768454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4530009e50 tx=0x7f4530009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=639221208f32e346 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.841+0000 7f453e7fc640 1 -- 172.21.15.63:0/1373768454 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4530031560 con 0x7f45400ffea0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.841+0000 7f453e7fc640 1 -- 172.21.15.63:0/1373768454 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f453002d050 con 0x7f45400ffea0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.841+0000 7f453e7fc640 1 -- 172.21.15.63:0/1373768454 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45300079a0 con 0x7f45400ffea0 2024-06-15T00:01:03.922 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 -- 172.21.15.63:0/1373768454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 msgr2=0x7f45401002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 --2- 172.21.15.63:0/1373768454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4530009e50 tx=0x7f4530009840 comp rx=0 tx=0).stop 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 -- 172.21.15.63:0/1373768454 shutdown_connections 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 --2- 172.21.15.63:0/1373768454 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f45401002b0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 -- 172.21.15.63:0/1373768454 >> 172.21.15.63:0/1373768454 conn(0x7f45400fb960 msgr2=0x7f45400fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 -- 172.21.15.63:0/1373768454 shutdown_connections 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 -- 172.21.15.63:0/1373768454 wait complete. 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.842+0000 7f4546082640 1 Processor -- start 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f4546082640 1 -- start start 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f4546082640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f4546082640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4540110dd0 con 0x7f45400ffea0 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f453f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f453f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47838/0 (socket says 172.21.15.63:47838) 2024-06-15T00:01:03.923 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f453f7fe640 1 -- 172.21.15.63:0/1271114270 learned_addr learned my addr 172.21.15.63:0/1271114270 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f453f7fe640 1 -- 172.21.15.63:0/1271114270 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4530009ae0 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.843+0000 7f453f7fe640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4530009560 tx=0x7f4530009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4530031bc0 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4530031d20 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4540110fd0 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f453003a530 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4540111440 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.844+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f4530038050 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.845+0000 7f4527fff640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 0x7f4528036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.845+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f4530066660 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.845+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4540111440 con 0x7f45400ffea0 2024-06-15T00:01:03.924 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.845+0000 7f453effd640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 0x7f4528036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.845+0000 7f453effd640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 0x7f4528036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4534006fd0 tx=0x7f4534006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:02.849+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4540111440 con 0x7f45400ffea0 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.024+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4540111440 con 0x7f4528034860 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.326+0000 7f4527fff640 1 -- 172.21.15.63:0/1271114270 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4540111440 con 0x7f4528034860 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 msgr2=0x7f4528036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 0x7f4528036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4534006fd0 tx=0x7f4534006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 msgr2=0x7f4540110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4530009560 tx=0x7f4530009840 comp rx=0 tx=0).stop 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 shutdown_connections 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f4528034860 0x7f4528036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 --2- 172.21.15.63:0/1271114270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45400ffea0 0x7f4540110890 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:03.925 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 >> 172.21.15.63:0/1271114270 conn(0x7f45400fb960 msgr2=0x7f45400fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:03.926 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 shutdown_connections 2024-06-15T00:01:03.926 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:03.328+0000 7f4546082640 1 -- 172.21.15.63:0/1271114270 wait complete. 2024-06-15T00:01:04.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:04 smithi063 ceph-mon[28882]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:04.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:04 smithi063 ceph-mon[28882]: Generating ssh key... 2024-06-15T00:01:04.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:04 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:04.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:04 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:05.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fwGN8sRqUy0kcQ2MJZ6BTPzbJFgGzOlh9HP/XZBesrXLlKyJXgKcn2AD6HH8/Uc4fjd3uE9/IFNJpi7maERSThmeRcbQgmG/gQXmZvAh8BDl7aIZgpCOWKoVpM5xJNQRlStHs0QHymBFMu0+6kRasqNWSEcm+mLYmqkXnZB8afcZf2cgaY+MrKPakh5vxRyirJHOySYlBUB73Uo/mW8mfBpjjtf1jkIWGftfd9iXfeDGw74Y2oUeNuwEn9oHfbzx9zbEQIT1UYs2/e6mP/MbrZzoUvvLsptZAhOgmQCmTVBSnm0P8twvFpJ91SQSMt/rFxddD0mk6clHyMgX1sbIGRSk5TOnMzwj8amlHJJmKj0bA8bDR4CISkNrponfhun6ymRALoium5gT13rn5fLqqi5Ti/lQMH3hG5j3ZeyAn81hLYtr/ute2XwCrLiDKLo5RigMtDQBxQxi3sx/ZiCiHLQB5XRJeVVnQNrDOtfp4ng/KRx0XTHDcgAIDSjrFQE= ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:01:05.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93ba52b640 1 Processor -- start 2024-06-15T00:01:05.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93ba52b640 1 -- start start 2024-06-15T00:01:05.095 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93ba52b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93ba52b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4100820 con 0x7f93b40ffed0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47848/0 (socket says 172.21.15.63:47848) 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.337+0000 7f93b3fff640 1 -- 172.21.15.63:0/610906270 learned_addr learned my addr 172.21.15.63:0/610906270 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.338+0000 7f93b3fff640 1 -- 172.21.15.63:0/610906270 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b4100960 con 0x7f93b40ffed0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.338+0000 7f93b3fff640 1 --2- 172.21.15.63:0/610906270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009840 tx=0x7f93a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d1b710dd4ce89d server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.338+0000 7f93b2ffd640 1 -- 172.21.15.63:0/610906270 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a4032440 con 0x7f93b40ffed0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.338+0000 7f93b2ffd640 1 -- 172.21.15.63:0/610906270 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f93a4032a10 con 0x7f93b40ffed0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93b2ffd640 1 -- 172.21.15.63:0/610906270 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a4031a00 con 0x7f93b40ffed0 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 -- 172.21.15.63:0/610906270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 msgr2=0x7f93b41002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 --2- 172.21.15.63:0/610906270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f93a4009840 tx=0x7f93a40091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:05.096 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 -- 172.21.15.63:0/610906270 shutdown_connections 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 --2- 172.21.15.63:0/610906270 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41002e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 -- 172.21.15.63:0/610906270 >> 172.21.15.63:0/610906270 conn(0x7f93b40fb990 msgr2=0x7f93b40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 -- 172.21.15.63:0/610906270 shutdown_connections 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.339+0000 7f93ba52b640 1 -- 172.21.15.63:0/610906270 wait complete. 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93ba52b640 1 Processor -- start 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93ba52b640 1 -- start start 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93ba52b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93ba52b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b4110de0 con 0x7f93b40ffed0 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93b3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47852/0 (socket says 172.21.15.63:47852) 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.340+0000 7f93b3fff640 1 -- 172.21.15.63:0/373378820 learned_addr learned my addr 172.21.15.63:0/373378820 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93b3fff640 1 -- 172.21.15.63:0/373378820 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a4009450 con 0x7f93b40ffed0 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93b3fff640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93a40313e0 tx=0x7f93a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:05.097 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a403d450 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f93a403da20 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b4110fe0 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93a403b8d0 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.341+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b4111450 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.342+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f93a4039070 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.342+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b404b480 con 0x7f93b40ffed0 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.342+0000 7f93b0ff9640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 0x7f939c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:05.098 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.342+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f93a4067550 con 0x7f93b40ffed0 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.342+0000 7f93b37fe640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 0x7f939c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.343+0000 7f93b37fe640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 0x7f939c036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f93a8006fd0 tx=0x7f93a8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.347+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f93a4046400 con 0x7f93b40ffed0 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.522+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f93b4111450 con 0x7f939c034860 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.522+0000 7f93b0ff9640 1 -- 172.21.15.63:0/373378820 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f93b4111450 con 0x7f939c034860 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 msgr2=0x7f939c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 0x7f939c036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f93a8006fd0 tx=0x7f93a8006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 msgr2=0x7f93b41108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f93a40313e0 tx=0x7f93a4008040 comp rx=0 tx=0).stop 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 shutdown_connections 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f939c034860 0x7f939c036d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 --2- 172.21.15.63:0/373378820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93b40ffed0 0x7f93b41108a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 >> 172.21.15.63:0/373378820 conn(0x7f93b40fb990 msgr2=0x7f93b40fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.524+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 shutdown_connections 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:04.525+0000 7f93ba52b640 1 -- 172.21.15.63:0/373378820 wait complete. 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:Adding key to root@localhost authorized_keys... 2024-06-15T00:01:05.100 INFO:teuthology.orchestra.run.smithi063.stdout:Adding host smithi063... 2024-06-15T00:01:05.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:05 smithi063 ceph-mon[28882]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:06.436 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:06 smithi063 ceph-mon[28882]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:07.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:07 smithi063 ceph-mon[28882]: Deploying cephadm binary to smithi063 2024-06-15T00:01:09.778 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Added host 'smithi063' with addr '172.21.15.63' 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.482+0000 7efce5930640 1 Processor -- start 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.482+0000 7efce5930640 1 -- start start 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efce5930640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efce5930640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce00780c0 con 0x7efce0077cb0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efcdeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efcdeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47858/0 (socket says 172.21.15.63:47858) 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efcdeffd640 1 -- 172.21.15.63:0/173179120 learned_addr learned my addr 172.21.15.63:0/173179120 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.483+0000 7efcdeffd640 1 -- 172.21.15.63:0/173179120 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce00767c0 con 0x7efce0077cb0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.484+0000 7efcdeffd640 1 --2- 172.21.15.63:0/173179120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009840 tx=0x7efcd00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c639ea2cef9f4f3 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.484+0000 7efcddffb640 1 -- 172.21.15.63:0/173179120 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd0032440 con 0x7efce0077cb0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.484+0000 7efcddffb640 1 -- 172.21.15.63:0/173179120 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7efcd0032a10 con 0x7efce0077cb0 2024-06-15T00:01:09.779 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.484+0000 7efcddffb640 1 -- 172.21.15.63:0/173179120 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd0031a00 con 0x7efce0077cb0 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 -- 172.21.15.63:0/173179120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 msgr2=0x7efce0076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 --2- 172.21.15.63:0/173179120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efcd0009840 tx=0x7efcd00091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 -- 172.21.15.63:0/173179120 shutdown_connections 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 --2- 172.21.15.63:0/173179120 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0076280 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 -- 172.21.15.63:0/173179120 >> 172.21.15.63:0/173179120 conn(0x7efce00fd330 msgr2=0x7efce00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 -- 172.21.15.63:0/173179120 shutdown_connections 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 -- 172.21.15.63:0/173179120 wait complete. 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.485+0000 7efce5930640 1 Processor -- start 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efce5930640 1 -- start start 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efce5930640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efce5930640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efce0112fe0 con 0x7efce0077cb0 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcdeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:09.780 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcdeffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47862/0 (socket says 172.21.15.63:47862) 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcdeffd640 1 -- 172.21.15.63:0/1069954438 learned_addr learned my addr 172.21.15.63:0/1069954438 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcdeffd640 1 -- 172.21.15.63:0/1069954438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd0009450 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcdeffd640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efcd00098a0 tx=0x7efcd0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd003d450 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7efcd003da20 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce01131e0 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.486+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcd003b860 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.487+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce0113650 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.487+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7efcd0039070 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.487+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce004b480 con 0x7efce0077cb0 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.488+0000 7efcc77fe640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 0x7efcc8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.488+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7efcd0067700 con 0x7efce0077cb0 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.488+0000 7efcde7fc640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 0x7efcc8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.488+0000 7efcde7fc640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 0x7efcc8036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcd4006fd0 tx=0x7efcd4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.492+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efcd004e940 con 0x7efce0077cb0 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:05.670+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi063", "addr": "172.21.15.63", "target": ["mon-mgr", ""]}) v1 -- 0x7efce0113650 con 0x7efcc8034860 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.200+0000 7efcc77fe640 1 -- 172.21.15.63:0/1069954438 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7efce0113650 con 0x7efcc8034860 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 msgr2=0x7efcc8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 0x7efcc8036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efcd4006fd0 tx=0x7efcd4006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 msgr2=0x7efce0112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7efcd00098a0 tx=0x7efcd0008040 comp rx=0 tx=0).stop 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 shutdown_connections 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7efcc8034860 0x7efcc8036d10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:09.782 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 --2- 172.21.15.63:0/1069954438 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efce0077cb0 0x7efce0112aa0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:09.783 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.202+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 >> 172.21.15.63:0/1069954438 conn(0x7efce00fd330 msgr2=0x7efce00fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:09.783 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.203+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 shutdown_connections 2024-06-15T00:01:09.783 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:09.203+0000 7efce5930640 1 -- 172.21.15.63:0/1069954438 wait complete. 2024-06-15T00:01:09.783 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying mon service with default placement... 2024-06-15T00:01:10.358 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:10 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:10.359 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:10 smithi063 ceph-mon[28882]: Added host smithi063 2024-06-15T00:01:10.359 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:10 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:01:10.979 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.166+0000 7ff64ad5a640 1 Processor -- start 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.166+0000 7ff64ad5a640 1 -- start start 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.166+0000 7ff64ad5a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.166+0000 7ff64ad5a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6440780c0 con 0x7ff644077cb0 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.167+0000 7ff648af6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.167+0000 7ff648af6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58236/0 (socket says 172.21.15.63:58236) 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.167+0000 7ff648af6640 1 -- 172.21.15.63:0/1519393612 learned_addr learned my addr 172.21.15.63:0/1519393612 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.167+0000 7ff648af6640 1 -- 172.21.15.63:0/1519393612 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6440767c0 con 0x7ff644077cb0 2024-06-15T00:01:10.980 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff648af6640 1 --2- 172.21.15.63:0/1519393612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff638009840 tx=0x7ff6380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec1571953755d85e server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff6437fe640 1 -- 172.21.15.63:0/1519393612 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff638032440 con 0x7ff644077cb0 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff6437fe640 1 -- 172.21.15.63:0/1519393612 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff638032a10 con 0x7ff644077cb0 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff6437fe640 1 -- 172.21.15.63:0/1519393612 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff638031a00 con 0x7ff644077cb0 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 -- 172.21.15.63:0/1519393612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 msgr2=0x7ff644076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/1519393612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff638009840 tx=0x7ff6380091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 -- 172.21.15.63:0/1519393612 shutdown_connections 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/1519393612 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644076280 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 -- 172.21.15.63:0/1519393612 >> 172.21.15.63:0/1519393612 conn(0x7ff6440fd330 msgr2=0x7ff6440ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 -- 172.21.15.63:0/1519393612 shutdown_connections 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.168+0000 7ff64ad5a640 1 -- 172.21.15.63:0/1519393612 wait complete. 2024-06-15T00:01:10.981 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff64ad5a640 1 Processor -- start 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff64ad5a640 1 -- start start 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff64ad5a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff64ad5a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff644112fe0 con 0x7ff644077cb0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff648af6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff648af6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58248/0 (socket says 172.21.15.63:58248) 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.169+0000 7ff648af6640 1 -- 172.21.15.63:0/2750847618 learned_addr learned my addr 172.21.15.63:0/2750847618 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff648af6640 1 -- 172.21.15.63:0/2750847618 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff638009450 con 0x7ff644077cb0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff648af6640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff6380098a0 tx=0x7ff638008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63803d450 con 0x7ff644077cb0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff63803da20 con 0x7ff644077cb0 2024-06-15T00:01:10.982 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6441131e0 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff63803b860 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.170+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff644113650 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.171+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff638039070 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.171+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff64404b480 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.171+0000 7ff6417fa640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 0x7ff634036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.171+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff638067570 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.171+0000 7ff643fff640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 0x7ff634036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.172+0000 7ff643fff640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 0x7ff634036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.176+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff63803ad50 con 0x7ff644077cb0 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.359+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7ff644113650 con 0x7ff634034860 2024-06-15T00:01:10.983 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.364+0000 7ff6417fa640 1 -- 172.21.15.63:0/2750847618 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff644113650 con 0x7ff634034860 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 msgr2=0x7ff634036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 0x7ff634036d10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff630006fd0 tx=0x7ff630006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 msgr2=0x7ff644112aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff6380098a0 tx=0x7ff638008040 comp rx=0 tx=0).stop 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 shutdown_connections 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff634034860 0x7ff634036d10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 --2- 172.21.15.63:0/2750847618 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff644077cb0 0x7ff644112aa0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 >> 172.21.15.63:0/2750847618 conn(0x7ff6440fd330 msgr2=0x7ff6440fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 shutdown_connections 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:10.366+0000 7ff64ad5a640 1 -- 172.21.15.63:0/2750847618 wait complete. 2024-06-15T00:01:10.984 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying mgr service with default placement... 2024-06-15T00:01:11.540 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:11 smithi063 ceph-mon[28882]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:11.540 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:11 smithi063 ceph-mon[28882]: Saving service mon spec with placement count:5 2024-06-15T00:01:11.540 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:11 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcfe29e640 1 Processor -- start 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcfe29e640 1 -- start start 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcfe29e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcfe29e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf8070c30 con 0x7fbcf8103110 2024-06-15T00:01:12.163 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcf7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcf7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58258/0 (socket says 172.21.15.63:58258) 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcf7fff640 1 -- 172.21.15.63:0/4240308537 learned_addr learned my addr 172.21.15.63:0/4240308537 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.348+0000 7fbcf7fff640 1 -- 172.21.15.63:0/4240308537 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcf8103a60 con 0x7fbcf8103110 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.349+0000 7fbcf7fff640 1 --2- 172.21.15.63:0/4240308537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009840 tx=0x7fbce80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baac9e9a4b11b4d5 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.349+0000 7fbcf6ffd640 1 -- 172.21.15.63:0/4240308537 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce8032440 con 0x7fbcf8103110 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.349+0000 7fbcf6ffd640 1 -- 172.21.15.63:0/4240308537 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbce8032a10 con 0x7fbcf8103110 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.349+0000 7fbcf6ffd640 1 -- 172.21.15.63:0/4240308537 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce8031a00 con 0x7fbcf8103110 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 -- 172.21.15.63:0/4240308537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 msgr2=0x7fbcf8103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/4240308537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fbce8009840 tx=0x7fbce80091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 -- 172.21.15.63:0/4240308537 shutdown_connections 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/4240308537 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8103520 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:12.164 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 -- 172.21.15.63:0/4240308537 >> 172.21.15.63:0/4240308537 conn(0x7fbcf80fb990 msgr2=0x7fbcf80fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 -- 172.21.15.63:0/4240308537 shutdown_connections 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 -- 172.21.15.63:0/4240308537 wait complete. 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.350+0000 7fbcfe29e640 1 Processor -- start 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcfe29e640 1 -- start start 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcfe29e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcfe29e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcf8102d60 con 0x7fbcf8103110 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcf7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcf7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58274/0 (socket says 172.21.15.63:58274) 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcf7fff640 1 -- 172.21.15.63:0/2756968906 learned_addr learned my addr 172.21.15.63:0/2756968906 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcf7fff640 1 -- 172.21.15.63:0/2756968906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbce8009450 con 0x7fbcf8103110 2024-06-15T00:01:12.165 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.351+0000 7fbcf7fff640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbce80098a0 tx=0x7fbce8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.352+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce8032830 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.352+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbce803d8c0 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.352+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbce803b7b0 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.352+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcf8100df0 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.352+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcf8101260 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fbce8039070 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf804b480 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcf4ff9640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 0x7fbce0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbce8067610 con 0x7fbcf8103110 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcf77fe640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 0x7fbce0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:12.166 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.353+0000 7fbcf77fe640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 0x7fbce0036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006fd0 tx=0x7fbcec006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.357+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbce803c590 con 0x7fbcf8103110 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.546+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fbcf8101a10 con 0x7fbce0034860 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.550+0000 7fbcf4ff9640 1 -- 172.21.15.63:0/2756968906 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbcf8101a10 con 0x7fbce0034860 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 msgr2=0x7fbce0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 0x7fbce0036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fbcec006fd0 tx=0x7fbcec006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 msgr2=0x7fbcf8102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbce80098a0 tx=0x7fbce8008040 comp rx=0 tx=0).stop 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 shutdown_connections 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fbce0034860 0x7fbce0036d10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 --2- 172.21.15.63:0/2756968906 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbcf8103110 0x7fbcf8102820 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 >> 172.21.15.63:0/2756968906 conn(0x7fbcf80fb990 msgr2=0x7fbcf80fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:12.167 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 shutdown_connections 2024-06-15T00:01:12.168 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:11.552+0000 7fbcfe29e640 1 -- 172.21.15.63:0/2756968906 wait complete. 2024-06-15T00:01:12.168 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying crash service with default placement... 2024-06-15T00:01:12.667 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:12 smithi063 ceph-mon[28882]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:12.667 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:12 smithi063 ceph-mon[28882]: Saving service mgr spec with placement count:2 2024-06-15T00:01:12.667 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:12 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:13.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-15T00:01:13.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f07318bd640 1 Processor -- start 2024-06-15T00:01:13.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f07318bd640 1 -- start start 2024-06-15T00:01:13.874 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f07318bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f07318bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0780a0 con 0x7f072c077c90 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f072affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f072affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58282/0 (socket says 172.21.15.63:58282) 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f072affd640 1 -- 172.21.15.63:0/3380872819 learned_addr learned my addr 172.21.15.63:0/3380872819 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.579+0000 7f072affd640 1 -- 172.21.15.63:0/3380872819 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0767a0 con 0x7f072c077c90 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.580+0000 7f072affd640 1 --2- 172.21.15.63:0/3380872819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f071c009840 tx=0x7f071c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d1e721f9f3dcd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.580+0000 7f0729ffb640 1 -- 172.21.15.63:0/3380872819 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f071c032440 con 0x7f072c077c90 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.580+0000 7f0729ffb640 1 -- 172.21.15.63:0/3380872819 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f071c032a10 con 0x7f072c077c90 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.580+0000 7f0729ffb640 1 -- 172.21.15.63:0/3380872819 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f071c031a00 con 0x7f072c077c90 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 -- 172.21.15.63:0/3380872819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 msgr2=0x7f072c076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 --2- 172.21.15.63:0/3380872819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f071c009840 tx=0x7f071c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 -- 172.21.15.63:0/3380872819 shutdown_connections 2024-06-15T00:01:13.875 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 --2- 172.21.15.63:0/3380872819 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c076260 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 -- 172.21.15.63:0/3380872819 >> 172.21.15.63:0/3380872819 conn(0x7f072c0fd330 msgr2=0x7f072c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 -- 172.21.15.63:0/3380872819 shutdown_connections 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.581+0000 7f07318bd640 1 -- 172.21.15.63:0/3380872819 wait complete. 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f07318bd640 1 Processor -- start 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f07318bd640 1 -- start start 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f07318bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f07318bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c112ff0 con 0x7f072c077c90 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f072affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f072affd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58290/0 (socket says 172.21.15.63:58290) 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f072affd640 1 -- 172.21.15.63:0/2935278202 learned_addr learned my addr 172.21.15.63:0/2935278202 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.582+0000 7f072affd640 1 -- 172.21.15.63:0/2935278202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071c009450 con 0x7f072c077c90 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f072affd640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f071c0098a0 tx=0x7f071c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f071c03d450 con 0x7f072c077c90 2024-06-15T00:01:13.876 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f071c03da20 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c1131f0 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f071c03b860 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.583+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c113660 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.584+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f071c039070 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.584+0000 7f07137fe640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 0x7f0714036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.584+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c04b480 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.584+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f071c067420 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.584+0000 7f072a7fc640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 0x7f0714036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.585+0000 7f072a7fc640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 0x7f0714036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.589+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f071c037020 con 0x7f072c077c90 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.776+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f072c113660 con 0x7f0714034860 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.781+0000 7f07137fe640 1 -- 172.21.15.63:0/2935278202 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f072c113660 con 0x7f0714034860 2024-06-15T00:01:13.877 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 msgr2=0x7f0714036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 0x7f0714036d10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0720006fd0 tx=0x7f0720006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 msgr2=0x7f072c112ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f071c0098a0 tx=0x7f071c008040 comp rx=0 tx=0).stop 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 shutdown_connections 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f0714034860 0x7f0714036d10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 --2- 172.21.15.63:0/2935278202 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f072c077c90 0x7f072c112ab0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 >> 172.21.15.63:0/2935278202 conn(0x7f072c0fd330 msgr2=0x7f072c0fdf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 shutdown_connections 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:12.783+0000 7f07318bd640 1 -- 172.21.15.63:0/2935278202 wait complete. 2024-06-15T00:01:13.878 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying prometheus service with default placement... 2024-06-15T00:01:13.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:13 smithi063 ceph-mon[28882]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:13.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:13 smithi063 ceph-mon[28882]: Saving service crash spec with placement * 2024-06-15T00:01:13.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:13 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:13.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:13 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:13.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:13 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:15.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-15T00:01:15.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.281+0000 7f480cfb9640 1 Processor -- start 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.281+0000 7f480cfb9640 1 -- start start 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.281+0000 7f480cfb9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.281+0000 7f480cfb9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4808102440 con 0x7f4808101af0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.282+0000 7f480659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.282+0000 7f480659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58294/0 (socket says 172.21.15.63:58294) 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.282+0000 7f480659c640 1 -- 172.21.15.63:0/3977741913 learned_addr learned my addr 172.21.15.63:0/3977741913 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.282+0000 7f480659c640 1 -- 172.21.15.63:0/3977741913 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4808102580 con 0x7f4808101af0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.282+0000 7f480659c640 1 --2- 172.21.15.63:0/3977741913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009840 tx=0x7f47f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c38155f5b27714c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480559a640 1 -- 172.21.15.63:0/3977741913 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f8032440 con 0x7f4808101af0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480559a640 1 -- 172.21.15.63:0/3977741913 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f8032a50 con 0x7f4808101af0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480559a640 1 -- 172.21.15.63:0/3977741913 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f8031aa0 con 0x7f4808101af0 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 -- 172.21.15.63:0/3977741913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 msgr2=0x7f4808101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:15.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3977741913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f47f8009840 tx=0x7f47f80091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 -- 172.21.15.63:0/3977741913 shutdown_connections 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3977741913 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808101f00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 -- 172.21.15.63:0/3977741913 >> 172.21.15.63:0/3977741913 conn(0x7f48080fd2f0 msgr2=0x7f48080ff700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.283+0000 7f480cfb9640 1 -- 172.21.15.63:0/3977741913 shutdown_connections 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480cfb9640 1 -- 172.21.15.63:0/3977741913 wait complete. 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480cfb9640 1 Processor -- start 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480cfb9640 1 -- start start 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480cfb9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480cfb9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4808110d80 con 0x7f4808101af0 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58306/0 (socket says 172.21.15.63:58306) 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.284+0000 7f480659c640 1 -- 172.21.15.63:0/3632184421 learned_addr learned my addr 172.21.15.63:0/3632184421 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:15.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f480659c640 1 -- 172.21.15.63:0/3632184421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f8009450 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f480659c640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47f8031780 tx=0x7f47f802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f803b440 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4808110f80 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47f803ba50 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47f803aab0 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.285+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4808111390 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.286+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f47f8039070 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.286+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480804b480 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.286+0000 7f47feffd640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 0x7f47f4036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.286+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f47f80670c0 con 0x7f4808101af0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.287+0000 7f4805d9b640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 0x7f47f4036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.287+0000 7f4805d9b640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 0x7f47f4036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006fd0 tx=0x7f47f0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:15.670 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.291+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f47f803ad20 con 0x7f4808101af0 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.495+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f48080fe5a0 con 0x7f47f4034810 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.499+0000 7f47feffd640 1 -- 172.21.15.63:0/3632184421 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f48080fe5a0 con 0x7f47f4034810 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 msgr2=0x7f47f4036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 0x7f47f4036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006fd0 tx=0x7f47f0006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 msgr2=0x7f4808110840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f47f8031780 tx=0x7f47f802c040 comp rx=0 tx=0).stop 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 shutdown_connections 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f47f4034810 0x7f47f4036cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 --2- 172.21.15.63:0/3632184421 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4808101af0 0x7f4808110840 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 >> 172.21.15.63:0/3632184421 conn(0x7f48080fd2f0 msgr2=0x7f48080fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 shutdown_connections 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:14.501+0000 7f480cfb9640 1 -- 172.21.15.63:0/3632184421 wait complete. 2024-06-15T00:01:15.671 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying grafana service with default placement... 2024-06-15T00:01:15.788 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:15 smithi063 ceph-mon[28882]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:15.788 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:15 smithi063 ceph-mon[28882]: Saving service prometheus spec with placement count:1 2024-06-15T00:01:15.789 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:15 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:15.789 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:15 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:16.917 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-15T00:01:16.917 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.085+0000 7f7958c83640 1 Processor -- start 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.085+0000 7f7958c83640 1 -- start start 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.085+0000 7f7958c83640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.085+0000 7f7958c83640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79541019b0 con 0x7f7954101060 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.085+0000 7f795259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.086+0000 7f795259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58316/0 (socket says 172.21.15.63:58316) 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.086+0000 7f795259c640 1 -- 172.21.15.63:0/2685116376 learned_addr learned my addr 172.21.15.63:0/2685116376 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.086+0000 7f795259c640 1 -- 172.21.15.63:0/2685116376 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7954101af0 con 0x7f7954101060 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f795259c640 1 --2- 172.21.15.63:0/2685116376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7944009840 tx=0x7f79440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405a6bb55f4209ed server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f795159a640 1 -- 172.21.15.63:0/2685116376 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7944032440 con 0x7f7954101060 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f795159a640 1 -- 172.21.15.63:0/2685116376 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7944032a50 con 0x7f7954101060 2024-06-15T00:01:16.918 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f795159a640 1 -- 172.21.15.63:0/2685116376 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7944031aa0 con 0x7f7954101060 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 -- 172.21.15.63:0/2685116376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 msgr2=0x7f7954101470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 --2- 172.21.15.63:0/2685116376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7944009840 tx=0x7f79440091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 -- 172.21.15.63:0/2685116376 shutdown_connections 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 --2- 172.21.15.63:0/2685116376 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954101470 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 -- 172.21.15.63:0/2685116376 >> 172.21.15.63:0/2685116376 conn(0x7f79540fb990 msgr2=0x7f79540fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 -- 172.21.15.63:0/2685116376 shutdown_connections 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.087+0000 7f7958c83640 1 -- 172.21.15.63:0/2685116376 wait complete. 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f7958c83640 1 Processor -- start 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f7958c83640 1 -- start start 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f7958c83640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f7958c83640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79541073d0 con 0x7f7954101060 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f795259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:16.919 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f795259c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58326/0 (socket says 172.21.15.63:58326) 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f795259c640 1 -- 172.21.15.63:0/2473523258 learned_addr learned my addr 172.21.15.63:0/2473523258 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.088+0000 7f795259c640 1 -- 172.21.15.63:0/2473523258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7944009450 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f795259c640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7944006f00 tx=0x7f794402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7944032830 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f794403b8f0 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f794403a8a0 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7954105460 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.089+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79541058d0 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f7944039070 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7942ffd640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 0x7f793c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7944066e30 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f795405ead0 con 0x7f7954101060 2024-06-15T00:01:16.920 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7951d9b640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 0x7f793c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.090+0000 7f7951d9b640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 0x7f793c036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7948006fd0 tx=0x7f7948006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.095+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7944037070 con 0x7f7954101060 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.284+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f7954105b10 con 0x7f793c034860 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.288+0000 7f7942ffd640 1 -- 172.21.15.63:0/2473523258 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f7954105b10 con 0x7f793c034860 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 msgr2=0x7f793c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 0x7f793c036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7948006fd0 tx=0x7f7948006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 msgr2=0x7f7954106e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7944006f00 tx=0x7f794402c040 comp rx=0 tx=0).stop 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 shutdown_connections 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f793c034860 0x7f793c036d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 --2- 172.21.15.63:0/2473523258 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7954101060 0x7f7954106e90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 >> 172.21.15.63:0/2473523258 conn(0x7f79540fb990 msgr2=0x7f79540fc880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 shutdown_connections 2024-06-15T00:01:16.921 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:16.290+0000 7f7958c83640 1 -- 172.21.15.63:0/2473523258 wait complete. 2024-06-15T00:01:16.922 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying node-exporter service with default placement... 2024-06-15T00:01:17.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:17 smithi063 ceph-mon[28882]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:17.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:17 smithi063 ceph-mon[28882]: Saving service grafana spec with placement count:1 2024-06-15T00:01:17.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:17 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:17.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:17 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:18.921 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:18 smithi063 ceph-mon[28882]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:18.922 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:18 smithi063 ceph-mon[28882]: Saving service node-exporter spec with placement * 2024-06-15T00:01:18.922 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:18 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:18.929 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-15T00:01:18.929 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46c1946640 1 Processor -- start 2024-06-15T00:01:18.929 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46c1946640 1 -- start start 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46c1946640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46c1946640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc102440 con 0x7f46bc101af0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46baffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46baffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58332/0 (socket says 172.21.15.63:58332) 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.334+0000 7f46baffd640 1 -- 172.21.15.63:0/3285183234 learned_addr learned my addr 172.21.15.63:0/3285183234 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.335+0000 7f46baffd640 1 -- 172.21.15.63:0/3285183234 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46bc102580 con 0x7f46bc101af0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.335+0000 7f46baffd640 1 --2- 172.21.15.63:0/3285183234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009e50 tx=0x7f46ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492998bd6f30b2a server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46b9ffb640 1 -- 172.21.15.63:0/3285183234 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac031560 con 0x7f46bc101af0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46b9ffb640 1 -- 172.21.15.63:0/3285183234 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46ac02d050 con 0x7f46bc101af0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46b9ffb640 1 -- 172.21.15.63:0/3285183234 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac007a20 con 0x7f46bc101af0 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 -- 172.21.15.63:0/3285183234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 msgr2=0x7f46bc101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:18.930 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 --2- 172.21.15.63:0/3285183234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46ac009e50 tx=0x7f46ac009840 comp rx=0 tx=0).stop 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 -- 172.21.15.63:0/3285183234 shutdown_connections 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 --2- 172.21.15.63:0/3285183234 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc101f00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 -- 172.21.15.63:0/3285183234 >> 172.21.15.63:0/3285183234 conn(0x7f46bc0fd330 msgr2=0x7f46bc0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 -- 172.21.15.63:0/3285183234 shutdown_connections 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.336+0000 7f46c1946640 1 -- 172.21.15.63:0/3285183234 wait complete. 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46c1946640 1 Processor -- start 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46c1946640 1 -- start start 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46c1946640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46c1946640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46bc078190 con 0x7f46bc101af0 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46baffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46baffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58338/0 (socket says 172.21.15.63:58338) 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.337+0000 7f46baffd640 1 -- 172.21.15.63:0/1422088169 learned_addr learned my addr 172.21.15.63:0/1422088169 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46baffd640 1 -- 172.21.15.63:0/1422088169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ac009ae0 con 0x7f46bc101af0 2024-06-15T00:01:18.931 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46baffd640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007700 tx=0x7f46ac009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac031a60 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46ac031bc0 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46bc076220 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46ac045570 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.338+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46bc076690 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.339+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f46ac038050 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.339+0000 7f46a37fe640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 0x7f46a4036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.339+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46bc04b480 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.339+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f46ac067020 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.339+0000 7f46ba7fc640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 0x7f46a4036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.340+0000 7f46ba7fc640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 0x7f46a4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46b0006fd0 tx=0x7f46b0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.344+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f46ac036070 con 0x7f46bc101af0 2024-06-15T00:01:18.932 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.540+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f46bc076d10 con 0x7f46a4034860 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.544+0000 7f46a37fe640 1 -- 172.21.15.63:0/1422088169 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f46bc076d10 con 0x7f46a4034860 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 msgr2=0x7f46a4036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 0x7f46a4036d10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f46b0006fd0 tx=0x7f46b0006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 msgr2=0x7f46bc077c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ac007700 tx=0x7f46ac009840 comp rx=0 tx=0).stop 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 shutdown_connections 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f46a4034860 0x7f46a4036d10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 --2- 172.21.15.63:0/1422088169 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f46bc101af0 0x7f46bc077c50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.546+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 >> 172.21.15.63:0/1422088169 conn(0x7f46bc0fd330 msgr2=0x7f46bc0fdf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.547+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 shutdown_connections 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:17.547+0000 7f46c1946640 1 -- 172.21.15.63:0/1422088169 wait complete. 2024-06-15T00:01:18.933 INFO:teuthology.orchestra.run.smithi063.stdout:Deploying alertmanager service with default placement... 2024-06-15T00:01:19.844 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:19 smithi063 ceph-mon[28882]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:19.844 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:19 smithi063 ceph-mon[28882]: from='mgr.14118 172.21.15.63:0/3831755581' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:20.110 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.354+0000 7fee332bd640 1 Processor -- start 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.354+0000 7fee332bd640 1 -- start start 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.354+0000 7fee332bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.354+0000 7fee332bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c102990 con 0x7fee2c102040 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.355+0000 7fee31059640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.355+0000 7fee31059640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40282/0 (socket says 172.21.15.63:40282) 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.355+0000 7fee31059640 1 -- 172.21.15.63:0/3958066207 learned_addr learned my addr 172.21.15.63:0/3958066207 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.355+0000 7fee31059640 1 -- 172.21.15.63:0/3958066207 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee2c100650 con 0x7fee2c102040 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee31059640 1 --2- 172.21.15.63:0/3958066207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fee28009840 tx=0x7fee280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7560eabe8653c4 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee23fff640 1 -- 172.21.15.63:0/3958066207 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee28032440 con 0x7fee2c102040 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee23fff640 1 -- 172.21.15.63:0/3958066207 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee28032a50 con 0x7fee2c102040 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee23fff640 1 -- 172.21.15.63:0/3958066207 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee28031aa0 con 0x7fee2c102040 2024-06-15T00:01:20.111 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee332bd640 1 -- 172.21.15.63:0/3958066207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 msgr2=0x7fee2c102450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.356+0000 7fee332bd640 1 --2- 172.21.15.63:0/3958066207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fee28009840 tx=0x7fee280091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- 172.21.15.63:0/3958066207 shutdown_connections 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 --2- 172.21.15.63:0/3958066207 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c102450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- 172.21.15.63:0/3958066207 >> 172.21.15.63:0/3958066207 conn(0x7fee2c0fc1a0 msgr2=0x7fee2c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- 172.21.15.63:0/3958066207 shutdown_connections 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- 172.21.15.63:0/3958066207 wait complete. 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 Processor -- start 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- start start 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.357+0000 7fee332bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2c110db0 con 0x7fee2c102040 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee31059640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee31059640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40296/0 (socket says 172.21.15.63:40296) 2024-06-15T00:01:20.112 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee31059640 1 -- 172.21.15.63:0/735925715 learned_addr learned my addr 172.21.15.63:0/735925715 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee31059640 1 -- 172.21.15.63:0/735925715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009450 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee31059640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee28009420 tx=0x7fee2802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2803b440 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2c110fb0 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee2803ba50 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.358+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee2803aa20 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.359+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2c111420 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.359+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fee28039070 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.360+0000 7fee21ffb640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 0x7fee18036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.360+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fee28066eb0 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.360+0000 7fee30858640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 0x7fee18036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.360+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2c05ead0 con 0x7fee2c102040 2024-06-15T00:01:20.113 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.360+0000 7fee30858640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 0x7fee18036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.365+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fee2803f080 con 0x7fee2c102040 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.547+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fee2c0fd370 con 0x7fee18034810 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.552+0000 7fee21ffb640 1 -- 172.21.15.63:0/735925715 <== mgr.14118 v2:172.21.15.63:6800/55223302 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fee2c0fd370 con 0x7fee18034810 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 msgr2=0x7fee18036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 0x7fee18036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 msgr2=0x7fee2c110870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee28009420 tx=0x7fee2802c040 comp rx=0 tx=0).stop 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 shutdown_connections 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fee18034810 0x7fee18036cc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 --2- 172.21.15.63:0/735925715 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fee2c102040 0x7fee2c110870 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 >> 172.21.15.63:0/735925715 conn(0x7fee2c0fc1a0 msgr2=0x7fee2c0fcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:20.114 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 shutdown_connections 2024-06-15T00:01:20.115 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:19.555+0000 7fee332bd640 1 -- 172.21.15.63:0/735925715 wait complete. 2024-06-15T00:01:21.000 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:20 smithi063 ceph-mon[28882]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:21.000 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:20 smithi063 ceph-mon[28882]: Saving service alertmanager spec with placement count:1 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe801ab8640 1 Processor -- start 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe801ab8640 1 -- start start 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe801ab8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe801ab8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc070c30 con 0x7fe7fc1030f0 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40302/0 (socket says 172.21.15.63:40302) 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.513+0000 7fe7fb7fe640 1 -- 172.21.15.63:0/2567370656 learned_addr learned my addr 172.21.15.63:0/2567370656 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.514+0000 7fe7fb7fe640 1 -- 172.21.15.63:0/2567370656 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc103a40 con 0x7fe7fc1030f0 2024-06-15T00:01:21.243 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.514+0000 7fe7fb7fe640 1 --2- 172.21.15.63:0/2567370656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009e50 tx=0x7fe7ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b07ed421eb45c93 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe7fa7fc640 1 -- 172.21.15.63:0/2567370656 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec031560 con 0x7fe7fc1030f0 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe7fa7fc640 1 -- 172.21.15.63:0/2567370656 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7ec02d050 con 0x7fe7fc1030f0 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe7fa7fc640 1 -- 172.21.15.63:0/2567370656 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec007a20 con 0x7fe7fc1030f0 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 -- 172.21.15.63:0/2567370656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 msgr2=0x7fe7fc103500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 --2- 172.21.15.63:0/2567370656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec009e50 tx=0x7fe7ec009840 comp rx=0 tx=0).stop 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 -- 172.21.15.63:0/2567370656 shutdown_connections 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 --2- 172.21.15.63:0/2567370656 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc103500 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 -- 172.21.15.63:0/2567370656 >> 172.21.15.63:0/2567370656 conn(0x7fe7fc0fb990 msgr2=0x7fe7fc0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 -- 172.21.15.63:0/2567370656 shutdown_connections 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.515+0000 7fe801ab8640 1 -- 172.21.15.63:0/2567370656 wait complete. 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe801ab8640 1 Processor -- start 2024-06-15T00:01:21.244 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe801ab8640 1 -- start start 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe801ab8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe801ab8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc110e10 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe7fb7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40308/0 (socket says 172.21.15.63:40308) 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe7fb7fe640 1 -- 172.21.15.63:0/3026438558 learned_addr learned my addr 172.21.15.63:0/3026438558 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.516+0000 7fe7fb7fe640 1 -- 172.21.15.63:0/3026438558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7ec009ae0 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe7fb7fe640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec007700 tx=0x7fe7ec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec031a60 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc111010 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe7ec031bc0 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7ec045570 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.517+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc111480 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fe7ec038050 con 0x7fe7fc1030f0 2024-06-15T00:01:21.245 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe7f3fff640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 0x7fe7e8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc04b480 con 0x7fe7fc1030f0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe7ec066d60 con 0x7fe7fc1030f0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe7faffd640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 0x7fe7e8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.518+0000 7fe7faffd640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 0x7fe7e8036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.522+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe7ec041080 con 0x7fe7fc1030f0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.699+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fe7fc100560 con 0x7fe7fc1030f0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.701+0000 7fe7f3fff640 1 -- 172.21.15.63:0/3026438558 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fe7ec04b070 con 0x7fe7fc1030f0 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.703+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 msgr2=0x7fe7e8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.703+0000 7fe801ab8640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 0x7fe7e8036cc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe7e4006fd0 tx=0x7fe7e4006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.703+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 msgr2=0x7fe7fc1108d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:21.246 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.703+0000 7fe801ab8640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec007700 tx=0x7fe7ec009840 comp rx=0 tx=0).stop 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 shutdown_connections 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe7e8034810 0x7fe7e8036cc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 --2- 172.21.15.63:0/3026438558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe7fc1030f0 0x7fe7fc1108d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 >> 172.21.15.63:0/3026438558 conn(0x7fe7fc0fb990 msgr2=0x7fe7fc0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 shutdown_connections 2024-06-15T00:01:21.247 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:20.704+0000 7fe801ab8640 1 -- 172.21.15.63:0/3026438558 wait complete. 2024-06-15T00:01:21.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:21 smithi063 ceph-mon[28882]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:21.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:21 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3026438558' entity='client.admin' 2024-06-15T00:01:22.375 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.622+0000 7ff6e5c08640 1 Processor -- start 2024-06-15T00:01:22.375 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.622+0000 7ff6e5c08640 1 -- start start 2024-06-15T00:01:22.375 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6e5c08640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6e5c08640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e00780e0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40312/0 (socket says 172.21.15.63:40312) 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6df7fe640 1 -- 172.21.15.63:0/2039735168 learned_addr learned my addr 172.21.15.63:0/2039735168 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.623+0000 7ff6df7fe640 1 -- 172.21.15.63:0/2039735168 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e00767e0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.624+0000 7ff6df7fe640 1 --2- 172.21.15.63:0/2039735168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009840 tx=0x7ff6d00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9b0766b999e9cf2 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.624+0000 7ff6de7fc640 1 -- 172.21.15.63:0/2039735168 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d0032440 con 0x7ff6e0077cd0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.624+0000 7ff6de7fc640 1 -- 172.21.15.63:0/2039735168 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6d0032a50 con 0x7ff6e0077cd0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.624+0000 7ff6de7fc640 1 -- 172.21.15.63:0/2039735168 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d0031aa0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.376 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2039735168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 msgr2=0x7ff6e00762a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2039735168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0009840 tx=0x7ff6d00091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2039735168 shutdown_connections 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2039735168 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e00762a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2039735168 >> 172.21.15.63:0/2039735168 conn(0x7ff6e00fd330 msgr2=0x7ff6e00ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2039735168 shutdown_connections 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2039735168 wait complete. 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 Processor -- start 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.625+0000 7ff6e5c08640 1 -- start start 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6e5c08640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6e5c08640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e0112fc0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.377 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40320/0 (socket says 172.21.15.63:40320) 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6df7fe640 1 -- 172.21.15.63:0/2642920418 learned_addr learned my addr 172.21.15.63:0/2642920418 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6df7fe640 1 -- 172.21.15.63:0/2642920418 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d0009450 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6df7fe640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0031780 tx=0x7ff6d002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d003b440 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6d003ba50 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e01131c0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.626+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6d003a980 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.627+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e0113630 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.627+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff6d0039070 con 0x7ff6e0077cd0 2024-06-15T00:01:22.378 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.628+0000 7ff6d7fff640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 0x7ff6cc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.628+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff6d0066d70 con 0x7ff6e0077cd0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.628+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e0113630 con 0x7ff6e0077cd0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.628+0000 7ff6deffd640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 0x7ff6cc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.628+0000 7ff6deffd640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 0x7ff6cc036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.632+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff6e0113630 con 0x7ff6e0077cd0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.805+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff6e0113630 con 0x7ff6e0077cd0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.809+0000 7ff6d7fff640 1 -- 172.21.15.63:0/2642920418 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff6d00650c0 con 0x7ff6e0077cd0 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 msgr2=0x7ff6cc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 0x7ff6cc036cc0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8006fd0 tx=0x7ff6c8006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 msgr2=0x7ff6e0112a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff6d0031780 tx=0x7ff6d002c040 comp rx=0 tx=0).stop 2024-06-15T00:01:22.379 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 shutdown_connections 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7ff6cc034810 0x7ff6cc036cc0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 --2- 172.21.15.63:0/2642920418 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff6e0077cd0 0x7ff6e0112a80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 >> 172.21.15.63:0/2642920418 conn(0x7ff6e00fd330 msgr2=0x7ff6e00fdf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 shutdown_connections 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:21.811+0000 7ff6e5c08640 1 -- 172.21.15.63:0/2642920418 wait complete. 2024-06-15T00:01:22.380 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling the dashboard module... 2024-06-15T00:01:23.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:22 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2642920418' entity='client.admin' 2024-06-15T00:01:24.098 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:23 smithi063 ceph-mon[28882]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:24.098 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:23 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3299404193' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-15T00:01:24.430 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e74ce1640 1 Processor -- start 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e74ce1640 1 -- start start 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e74ce1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e74ce1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70100820 con 0x7f8e700ffed0 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e6e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e6e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40328/0 (socket says 172.21.15.63:40328) 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.780+0000 7f8e6e59c640 1 -- 172.21.15.63:0/2578218485 learned_addr learned my addr 172.21.15.63:0/2578218485 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.781+0000 7f8e6e59c640 1 -- 172.21.15.63:0/2578218485 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e70100960 con 0x7f8e700ffed0 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.781+0000 7f8e6e59c640 1 --2- 172.21.15.63:0/2578218485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009840 tx=0x7f8e600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ad70383c8f5b8f server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e6d59a640 1 -- 172.21.15.63:0/2578218485 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e60032440 con 0x7f8e700ffed0 2024-06-15T00:01:24.431 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e6d59a640 1 -- 172.21.15.63:0/2578218485 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e60032a50 con 0x7f8e700ffed0 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e6d59a640 1 -- 172.21.15.63:0/2578218485 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e60031aa0 con 0x7f8e700ffed0 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 -- 172.21.15.63:0/2578218485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 msgr2=0x7f8e701002e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/2578218485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009840 tx=0x7f8e600091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 -- 172.21.15.63:0/2578218485 shutdown_connections 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/2578218485 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e701002e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 -- 172.21.15.63:0/2578218485 >> 172.21.15.63:0/2578218485 conn(0x7f8e700fb990 msgr2=0x7f8e700fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.782+0000 7f8e74ce1640 1 -- 172.21.15.63:0/2578218485 shutdown_connections 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e74ce1640 1 -- 172.21.15.63:0/2578218485 wait complete. 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e74ce1640 1 Processor -- start 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e74ce1640 1 -- start start 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e74ce1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:24.432 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e74ce1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e70110d60 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e6e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e6e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40330/0 (socket says 172.21.15.63:40330) 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.783+0000 7f8e6e59c640 1 -- 172.21.15.63:0/3299404193 learned_addr learned my addr 172.21.15.63:0/3299404193 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e6e59c640 1 -- 172.21.15.63:0/3299404193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e60009450 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e6e59c640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009420 tx=0x7f8e6002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6003b440 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e6003ba50 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e6003aa20 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e70110f60 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.784+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e701113d0 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.785+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8e60039070 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.785+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e7004b480 con 0x7f8e700ffed0 2024-06-15T00:01:24.433 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.785+0000 7f8e5effd640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 0x7f8e58036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.785+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8e60066ec0 con 0x7f8e700ffed0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.785+0000 7f8e6dd9b640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 0x7f8e58036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.786+0000 7f8e6dd9b640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 0x7f8e58036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e64006fd0 tx=0x7f8e64006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:22.790+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8e60039500 con 0x7f8e700ffed0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.009+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f8e701113d0 con 0x7f8e700ffed0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.812+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f8e60037210 con 0x7f8e700ffed0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.813+0000 7f8e5effd640 1 -- 172.21.15.63:0/3299404193 <== mon.0 v2:172.21.15.63:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f8e60043820 con 0x7f8e700ffed0 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 msgr2=0x7f8e58036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 0x7f8e58036cc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8e64006fd0 tx=0x7f8e64006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 msgr2=0x7f8e70110820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:24.434 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e60009420 tx=0x7f8e6002c040 comp rx=0 tx=0).stop 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 shutdown_connections 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f8e58034810 0x7f8e58036cc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 --2- 172.21.15.63:0/3299404193 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8e700ffed0 0x7f8e70110820 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 >> 172.21.15.63:0/3299404193 conn(0x7f8e700fb990 msgr2=0x7f8e700fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 shutdown_connections 2024-06-15T00:01:24.435 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:23.816+0000 7f8e74ce1640 1 -- 172.21.15.63:0/3299404193 wait complete. 2024-06-15T00:01:25.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3299404193' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-15T00:01:25.052 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:24 smithi063 ceph-mon[28882]: mgrmap e9: smithi063.dhizav(active, since 25s) 2024-06-15T00:01:25.663 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "active_name": "smithi063.dhizav", 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.846+0000 7fe10adc9640 1 Processor -- start 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.846+0000 7fe10adc9640 1 -- start start 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe10adc9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe10adc9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1041023b0 con 0x7fe104101a60 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe108b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:25.664 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe108b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40360/0 (socket says 172.21.15.63:40360) 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe108b65640 1 -- 172.21.15.63:0/3182098935 learned_addr learned my addr 172.21.15.63:0/3182098935 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.847+0000 7fe108b65640 1 -- 172.21.15.63:0/3182098935 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1041024f0 con 0x7fe104101a60 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.848+0000 7fe108b65640 1 --2- 172.21.15.63:0/3182098935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009840 tx=0x7fe0f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c457223157b24f0c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.848+0000 7fe1037fe640 1 -- 172.21.15.63:0/3182098935 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f8032440 con 0x7fe104101a60 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.848+0000 7fe1037fe640 1 -- 172.21.15.63:0/3182098935 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f8032a50 con 0x7fe104101a60 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.848+0000 7fe1037fe640 1 -- 172.21.15.63:0/3182098935 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f8031aa0 con 0x7fe104101a60 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- 172.21.15.63:0/3182098935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 msgr2=0x7fe104101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 --2- 172.21.15.63:0/3182098935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009840 tx=0x7fe0f80091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- 172.21.15.63:0/3182098935 shutdown_connections 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 --2- 172.21.15.63:0/3182098935 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104101e70 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- 172.21.15.63:0/3182098935 >> 172.21.15.63:0/3182098935 conn(0x7fe1040fd280 msgr2=0x7fe1040ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- 172.21.15.63:0/3182098935 shutdown_connections 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- 172.21.15.63:0/3182098935 wait complete. 2024-06-15T00:01:25.665 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 Processor -- start 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.849+0000 7fe10adc9640 1 -- start start 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe10adc9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe10adc9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe104110d40 con 0x7fe104101a60 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe108b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe108b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40370/0 (socket says 172.21.15.63:40370) 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe108b65640 1 -- 172.21.15.63:0/2179107076 learned_addr learned my addr 172.21.15.63:0/2179107076 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe108b65640 1 -- 172.21.15.63:0/2179107076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f8009450 con 0x7fe104101a60 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe108b65640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009420 tx=0x7fe0f802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.850+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f803b440 con 0x7fe104101a60 2024-06-15T00:01:25.666 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.851+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0f803ba50 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.851+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe104110f40 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.851+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0f803aa20 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.851+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1041113b0 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.851+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fe0f8039070 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.852+0000 7fe1017fa640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 0x7fe0f4036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.852+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10404b480 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.852+0000 7fe103fff640 1 -- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 msgr2=0x7fe0f4036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.852+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe0f8067040 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.852+0000 7fe103fff640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 0x7fe0f4036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:24.856+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe0f8030700 con 0x7fe104101a60 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.053+0000 7fe103fff640 1 -- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 msgr2=0x7fe0f4036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:25.667 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.053+0000 7fe103fff640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 0x7fe0f4036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.075+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fe1041113b0 con 0x7fe104101a60 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.075+0000 7fe1017fa640 1 -- 172.21.15.63:0/2179107076 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fe0f8049070 con 0x7fe104101a60 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 msgr2=0x7fe0f4036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 0x7fe0f4036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 msgr2=0x7fe104110800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe0f8009420 tx=0x7fe0f802c040 comp rx=0 tx=0).stop 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 shutdown_connections 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7fe0f40348b0 0x7fe0f4036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 --2- 172.21.15.63:0/2179107076 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe104101a60 0x7fe104110800 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 >> 172.21.15.63:0/2179107076 conn(0x7fe1040fd280 msgr2=0x7fe1040fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:25.668 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 shutdown_connections 2024-06-15T00:01:25.669 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:25.077+0000 7fe10adc9640 1 -- 172.21.15.63:0/2179107076 wait complete. 2024-06-15T00:01:25.669 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for the mgr to restart... 2024-06-15T00:01:25.669 INFO:teuthology.orchestra.run.smithi063.stdout:Waiting for mgr epoch 9... 2024-06-15T00:01:25.911 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:25 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2179107076' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-15T00:01:28.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: Active manager daemon smithi063.dhizav restarted 2024-06-15T00:01:28.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: Activating manager daemon smithi063.dhizav 2024-06-15T00:01:28.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: osdmap e3: 0 total, 0 up, 0 in 2024-06-15T00:01:28.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: mgrmap e10: smithi063.dhizav(active, starting, since 0.00318573s) 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr metadata", "who": "smithi063.dhizav", "id": "smithi063.dhizav"}]: dispatch 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: Manager daemon smithi063.dhizav is now available 2024-06-15T00:01:28.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:01:28.682 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:01:28.682 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:01:28.682 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/trash_purge_schedule"}]: dispatch 2024-06-15T00:01:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:01:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-15T00:01:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-15T00:01:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:01:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.104+0000 7f955947a640 1 Processor -- start 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.104+0000 7f955947a640 1 -- start start 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.104+0000 7f955947a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.104+0000 7f955947a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95541012c0 con 0x7f9554100970 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.105+0000 7f9552ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.105+0000 7f9552ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40382/0 (socket says 172.21.15.63:40382) 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.105+0000 7f9552ffd640 1 -- 172.21.15.63:0/409634606 learned_addr learned my addr 172.21.15.63:0/409634606 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.105+0000 7f9552ffd640 1 -- 172.21.15.63:0/409634606 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554101400 con 0x7f9554100970 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.106+0000 7f9552ffd640 1 --2- 172.21.15.63:0/409634606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9544009840 tx=0x7f95440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46712e2747569b20 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.106+0000 7f9551ffb640 1 -- 172.21.15.63:0/409634606 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9544032440 con 0x7f9554100970 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.106+0000 7f9551ffb640 1 -- 172.21.15.63:0/409634606 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9544032a50 con 0x7f9554100970 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.106+0000 7f9551ffb640 1 -- 172.21.15.63:0/409634606 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9544031aa0 con 0x7f9554100970 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 -- 172.21.15.63:0/409634606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 msgr2=0x7f9554100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:30.452 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 --2- 172.21.15.63:0/409634606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9544009840 tx=0x7f95440091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 -- 172.21.15.63:0/409634606 shutdown_connections 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 --2- 172.21.15.63:0/409634606 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554100d80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 -- 172.21.15.63:0/409634606 >> 172.21.15.63:0/409634606 conn(0x7f95540fc190 msgr2=0x7f95540fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 -- 172.21.15.63:0/409634606 shutdown_connections 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.107+0000 7f955947a640 1 -- 172.21.15.63:0/409634606 wait complete. 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f955947a640 1 Processor -- start 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f955947a640 1 -- start start 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f955947a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f955947a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9554112fa0 con 0x7f9554100970 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f9552ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f9552ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40388/0 (socket says 172.21.15.63:40388) 2024-06-15T00:01:30.453 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f9552ffd640 1 -- 172.21.15.63:0/4016634055 learned_addr learned my addr 172.21.15.63:0/4016634055 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f9552ffd640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9544009450 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.108+0000 7f9552ffd640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9544009420 tx=0x7f954402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.109+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954403b440 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.109+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f954403ba50 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.109+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95541131a0 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.109+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f954403aa20 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.109+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9554113610 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f9544039070 con 0x7f9554100970 2024-06-15T00:01:30.454 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f953b7fe640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9544066fa0 con 0x7f9554100970 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f955405ead0 con 0x7f953c0348b0 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f95527fc640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 msgr2=0x7f953c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.110+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.311+0000 7f95527fc640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 msgr2=0x7f953c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.311+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.712+0000 7f95527fc640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 msgr2=0x7f953c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:26.712+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:27.513+0000 7f95527fc640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 msgr2=0x7f953c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/55223302 2024-06-15T00:01:30.455 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:27.513+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:28.539+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7f9544066160 con 0x7f9554100970 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:28.539+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 msgr2=0x7f953c036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:28.539+0000 7f953b7fe640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.540+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f9544039320 con 0x7f9554100970 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.541+0000 7f953b7fe640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 0x7f953c03a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.541+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f955405ead0 con 0x7f953c037fe0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.542+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 0x7f953c03a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.543+0000 7f95527fc640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 0x7f953c03a3c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9548003940 tx=0x7f9548009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.543+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f955405ead0 con 0x7f953c037fe0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f955404b480 con 0x7f953c037fe0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f953b7fe640 1 -- 172.21.15.63:0/4016634055 <== mgr.14160 v2:172.21.15.63:6800/3768764057 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f955404b480 con 0x7f953c037fe0 2024-06-15T00:01:30.456 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 msgr2=0x7f953c03a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f955947a640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 0x7f953c03a3c0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f9548003940 tx=0x7f9548009350 comp rx=0 tx=0).stop 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 msgr2=0x7f9554112a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.549+0000 7f955947a640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9544009420 tx=0x7f954402c040 comp rx=0 tx=0).stop 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 shutdown_connections 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f953c037fe0 0x7f953c03a3c0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:6800/55223302,v1:172.21.15.63:6801/55223302] conn(0x7f953c0348b0 0x7f953c036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 --2- 172.21.15.63:0/4016634055 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9554100970 0x7f9554112a60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 >> 172.21.15.63:0/4016634055 conn(0x7f95540fc190 msgr2=0x7f95540fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 shutdown_connections 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:29.550+0000 7f955947a640 1 -- 172.21.15.63:0/4016634055 wait complete. 2024-06-15T00:01:30.457 INFO:teuthology.orchestra.run.smithi063.stdout:mgr epoch 9 is available 2024-06-15T00:01:30.458 INFO:teuthology.orchestra.run.smithi063.stdout:Generating a dashboard self-signed certificate... 2024-06-15T00:01:30.851 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:30 smithi063 ceph-mon[28882]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-15T00:01:30.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:30 smithi063 ceph-mon[28882]: mgrmap e11: smithi063.dhizav(active, since 1.00616s) 2024-06-15T00:01:30.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:30 smithi063 ceph-mon[28882]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-15T00:01:30.852 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:30 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:31.997 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.856+0000 7f134f8c0640 1 Processor -- start 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.856+0000 7f134f8c0640 1 -- start start 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.856+0000 7f134f8c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.856+0000 7f134f8c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13481023b0 con 0x7f1348101a60 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.857+0000 7f134d65c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.857+0000 7f134d65c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33102/0 (socket says 172.21.15.63:33102) 2024-06-15T00:01:31.998 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.857+0000 7f134d65c640 1 -- 172.21.15.63:0/1580115690 learned_addr learned my addr 172.21.15.63:0/1580115690 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.857+0000 7f134d65c640 1 -- 172.21.15.63:0/1580115690 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13481024f0 con 0x7f1348101a60 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.858+0000 7f134d65c640 1 --2- 172.21.15.63:0/1580115690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1344009840 tx=0x7f13440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f735a690e9c228a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.858+0000 7f133ffff640 1 -- 172.21.15.63:0/1580115690 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1344032440 con 0x7f1348101a60 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.858+0000 7f133ffff640 1 -- 172.21.15.63:0/1580115690 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1344032a50 con 0x7f1348101a60 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.858+0000 7f133ffff640 1 -- 172.21.15.63:0/1580115690 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1344031aa0 con 0x7f1348101a60 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 -- 172.21.15.63:0/1580115690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 msgr2=0x7f1348101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1580115690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1344009840 tx=0x7f13440091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 -- 172.21.15.63:0/1580115690 shutdown_connections 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1580115690 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348101e70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 -- 172.21.15.63:0/1580115690 >> 172.21.15.63:0/1580115690 conn(0x7f13480fd280 msgr2=0x7f13480ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 -- 172.21.15.63:0/1580115690 shutdown_connections 2024-06-15T00:01:31.999 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.859+0000 7f134f8c0640 1 -- 172.21.15.63:0/1580115690 wait complete. 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134f8c0640 1 Processor -- start 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134f8c0640 1 -- start start 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134f8c0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134f8c0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348078190 con 0x7f1348101a60 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134d65c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134d65c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33108/0 (socket says 172.21.15.63:33108) 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.860+0000 7f134d65c640 1 -- 172.21.15.63:0/1056184054 learned_addr learned my addr 172.21.15.63:0/1056184054 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f134d65c640 1 -- 172.21.15.63:0/1056184054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1344009450 con 0x7f1348101a60 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f134d65c640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1344009420 tx=0x7f134402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f134403b440 con 0x7f1348101a60 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f134403ba50 con 0x7f1348101a60 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348076220 con 0x7f1348101a60 2024-06-15T00:01:32.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f134403aa20 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.861+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348076690 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.862+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f1344039070 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.862+0000 7f133dffb640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 0x7f1334036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.862+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1344066ff0 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.862+0000 7f134ce5b640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 0x7f1334036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.863+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134804b480 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.863+0000 7f134ce5b640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 0x7f1334036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1338006fd0 tx=0x7f1338006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:30.869+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f134403f020 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.051+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f1348076c10 con 0x7f1334034790 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.115+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f1344092df0 con 0x7f1348101a60 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.202+0000 7f133dffb640 1 -- 172.21.15.63:0/1056184054 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f1348076c10 con 0x7f1334034790 2024-06-15T00:01:32.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 msgr2=0x7f1334036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 0x7f1334036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f1338006fd0 tx=0x7f1338006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 msgr2=0x7f1348077c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1344009420 tx=0x7f134402c040 comp rx=0 tx=0).stop 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 shutdown_connections 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f1334034790 0x7f1334036c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 --2- 172.21.15.63:0/1056184054 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1348101a60 0x7f1348077c50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 >> 172.21.15.63:0/1056184054 conn(0x7f13480fd280 msgr2=0x7f13480fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 shutdown_connections 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:31.205+0000 7f134f8c0640 1 -- 172.21.15.63:0/1056184054 wait complete. 2024-06-15T00:01:32.002 INFO:teuthology.orchestra.run.smithi063.stdout:Creating initial admin user... 2024-06-15T00:01:32.116 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:30] ENGINE Bus STARTING 2024-06-15T00:01:32.116 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:30] ENGINE Serving on https://172.21.15.63:7150 2024-06-15T00:01:32.116 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:30] ENGINE Bus STARTED 2024-06-15T00:01:32.116 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: [15/Jun/2024:00:01:30] ENGINE Error in HTTPServer.serve 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Traceback (most recent call last): 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._run(expiration_interval) 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s = self.context.wrap_socket( 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:01:32.117 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: return self.sslsocket_class._create( 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self.do_handshake() 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._sslobj.do_handshake() 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: mgrmap e12: smithi063.dhizav(active, since 2s) 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:32.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:31 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:33.017 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:32 smithi063 ceph-mon[28882]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$7yW8n/S13lyy.eh3b2wHGu2Tn8k19UzZHnuHX72nxUgzZY.63LBSW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718409693, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f284b9a5640 1 Processor -- start 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f284b9a5640 1 -- start start 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f284b9a5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f284b9a5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844104c00 con 0x7f2844075c50 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f2849741640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f2849741640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33120/0 (socket says 172.21.15.63:33120) 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f2849741640 1 -- 172.21.15.63:0/1329178060 learned_addr learned my addr 172.21.15.63:0/1329178060 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.818+0000 7f2849741640 1 -- 172.21.15.63:0/1329178060 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2844104d40 con 0x7f2844075c50 2024-06-15T00:01:33.851 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.819+0000 7f2849741640 1 --2- 172.21.15.63:0/1329178060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f283c009840 tx=0x7f283c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef29350e38134780 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.819+0000 7f283bfff640 1 -- 172.21.15.63:0/1329178060 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f283c032440 con 0x7f2844075c50 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.819+0000 7f283bfff640 1 -- 172.21.15.63:0/1329178060 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f283c032a50 con 0x7f2844075c50 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.819+0000 7f283bfff640 1 -- 172.21.15.63:0/1329178060 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f283c031ad0 con 0x7f2844075c50 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 -- 172.21.15.63:0/1329178060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 msgr2=0x7f28441046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 --2- 172.21.15.63:0/1329178060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f283c009840 tx=0x7f283c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 -- 172.21.15.63:0/1329178060 shutdown_connections 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 --2- 172.21.15.63:0/1329178060 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f28441046c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 -- 172.21.15.63:0/1329178060 >> 172.21.15.63:0/1329178060 conn(0x7f28440fc190 msgr2=0x7f28440fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 -- 172.21.15.63:0/1329178060 shutdown_connections 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.820+0000 7f284b9a5640 1 -- 172.21.15.63:0/1329178060 wait complete. 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f284b9a5640 1 Processor -- start 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f284b9a5640 1 -- start start 2024-06-15T00:01:33.852 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f284b9a5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f284b9a5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2844102530 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f2849741640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f2849741640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33128/0 (socket says 172.21.15.63:33128) 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f2849741640 1 -- 172.21.15.63:0/3752163475 learned_addr learned my addr 172.21.15.63:0/3752163475 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.821+0000 7f2849741640 1 -- 172.21.15.63:0/3752163475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c009450 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f2849741640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f283c0317b0 tx=0x7f283c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f283c0327b0 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f283c03b8e0 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28441005c0 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f283c03a780 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.822+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2844100a30 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.823+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f283c039040 con 0x7f2844075c50 2024-06-15T00:01:33.853 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.823+0000 7f2839ffb640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 0x7f2830036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.823+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f284404b480 con 0x7f2844075c50 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.823+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f283c066dd0 con 0x7f2844075c50 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.824+0000 7f2848f40640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 0x7f2830036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.824+0000 7f2848f40640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 0x7f2830036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2834006fd0 tx=0x7f2834006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:32.828+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f283c03f080 con 0x7f2844075c50 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.018+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f284405e8c0 con 0x7f28300348b0 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.267+0000 7f2839ffb640 1 -- 172.21.15.63:0/3752163475 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f284405e8c0 con 0x7f28300348b0 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 msgr2=0x7f2830036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 0x7f2830036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2834006fd0 tx=0x7f2834006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 msgr2=0x7f2844101ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f283c0317b0 tx=0x7f283c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.854 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 shutdown_connections 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f28300348b0 0x7f2830036d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 --2- 172.21.15.63:0/3752163475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2844075c50 0x7f2844101ff0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 >> 172.21.15.63:0/3752163475 conn(0x7f28440fc190 msgr2=0x7f28440fcd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 shutdown_connections 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:33.269+0000 7f284b9a5640 1 -- 172.21.15.63:0/3752163475 wait complete. 2024-06-15T00:01:33.855 INFO:teuthology.orchestra.run.smithi063.stdout:Fetching dashboard port number... 2024-06-15T00:01:34.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:34 smithi063 ceph-mon[28882]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:34.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:34 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:35.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stdout 8443 2024-06-15T00:01:35.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.227+0000 7f4a6fdbc640 1 Processor -- start 2024-06-15T00:01:35.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.227+0000 7f4a6fdbc640 1 -- start start 2024-06-15T00:01:35.000 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6fdbc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6fdbc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68104c20 con 0x7f4a68075c20 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6db58640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6db58640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33130/0 (socket says 172.21.15.63:33130) 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6db58640 1 -- 172.21.15.63:0/3586792835 learned_addr learned my addr 172.21.15.63:0/3586792835 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.228+0000 7f4a6db58640 1 -- 172.21.15.63:0/3586792835 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a68104d60 con 0x7f4a68075c20 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6db58640 1 --2- 172.21.15.63:0/3586792835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009e50 tx=0x7f4a58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62e1d27923fc6fa6 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6cb56640 1 -- 172.21.15.63:0/3586792835 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58031560 con 0x7f4a68075c20 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6cb56640 1 -- 172.21.15.63:0/3586792835 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a5802d050 con 0x7f4a68075c20 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6cb56640 1 -- 172.21.15.63:0/3586792835 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58007a20 con 0x7f4a68075c20 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3586792835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 msgr2=0x7f4a681046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.229+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3586792835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009e50 tx=0x7f4a58009840 comp rx=0 tx=0).stop 2024-06-15T00:01:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3586792835 shutdown_connections 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3586792835 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681046e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3586792835 >> 172.21.15.63:0/3586792835 conn(0x7f4a680fc150 msgr2=0x7f4a680fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3586792835 shutdown_connections 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3586792835 wait complete. 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 Processor -- start 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- start start 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.230+0000 7f4a6fdbc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a68110e30 con 0x7f4a68075c20 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6db58640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6db58640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33140/0 (socket says 172.21.15.63:33140) 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6db58640 1 -- 172.21.15.63:0/3317948420 learned_addr learned my addr 172.21.15.63:0/3317948420 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6db58640 1 -- 172.21.15.63:0/3317948420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a58009ae0 con 0x7f4a68075c20 2024-06-15T00:01:35.002 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6db58640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009e20 tx=0x7f4a58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58031de0 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a5803b8c0 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a68111030 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a58045770 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.231+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a681114a0 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.232+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f4a58038050 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.232+0000 7f4a667fc640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 0x7f4a50036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.232+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4a58067300 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.232+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a6805ead0 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.232+0000 7f4a6d357640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 0x7f4a50036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.233+0000 7f4a6d357640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 0x7f4a50036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006fd0 tx=0x7f4a5c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.237+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4a58041020 con 0x7f4a68075c20 2024-06-15T00:01:35.003 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.411+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f4a6804b480 con 0x7f4a68075c20 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.412+0000 7f4a667fc640 1 -- 172.21.15.63:0/3317948420 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f4a58036070 con 0x7f4a68075c20 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 msgr2=0x7f4a50036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 0x7f4a50036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c006fd0 tx=0x7f4a5c006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 msgr2=0x7f4a681108f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a58009e20 tx=0x7f4a58009840 comp rx=0 tx=0).stop 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 shutdown_connections 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4a50034860 0x7f4a50036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 --2- 172.21.15.63:0/3317948420 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4a68075c20 0x7f4a681108f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 >> 172.21.15.63:0/3317948420 conn(0x7f4a680fc150 msgr2=0x7f4a680fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 shutdown_connections 2024-06-15T00:01:35.004 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:34.414+0000 7f4a6fdbc640 1 -- 172.21.15.63:0/3317948420 wait complete. 2024-06-15T00:01:35.008 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:01:35.008 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout disabled 2024-06-15T00:01:35.016 INFO:teuthology.orchestra.run.smithi063.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:01:35.016 INFO:teuthology.orchestra.run.smithi063.stdout:systemctl: stdout inactive 2024-06-15T00:01:35.016 INFO:teuthology.orchestra.run.smithi063.stdout:firewalld.service is not enabled 2024-06-15T00:01:35.017 INFO:teuthology.orchestra.run.smithi063.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-15T00:01:35.018 INFO:teuthology.orchestra.run.smithi063.stdout:Ceph Dashboard is now available at: 2024-06-15T00:01:35.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:35.019 INFO:teuthology.orchestra.run.smithi063.stdout: URL: https://smithi063.front.sepia.ceph.com:8443/ 2024-06-15T00:01:35.019 INFO:teuthology.orchestra.run.smithi063.stdout: User: admin 2024-06-15T00:01:35.019 INFO:teuthology.orchestra.run.smithi063.stdout: Password: e2ewq0jhtj 2024-06-15T00:01:35.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:35.019 INFO:teuthology.orchestra.run.smithi063.stdout:Saving cluster configuration to /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config directory 2024-06-15T00:01:35.020 INFO:teuthology.orchestra.run.smithi063.stdout:Enabling autotune for osd_memory_target 2024-06-15T00:01:35.619 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:35 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3317948420' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-15T00:01:36.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.435+0000 7f8284943640 1 Processor -- start 2024-06-15T00:01:36.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f8284943640 1 -- start start 2024-06-15T00:01:36.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f8284943640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:36.215 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f8284943640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280104c00 con 0x7f8280075c40 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f827e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f827e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33146/0 (socket says 172.21.15.63:33146) 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f827e59c640 1 -- 172.21.15.63:0/3352170923 learned_addr learned my addr 172.21.15.63:0/3352170923 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.436+0000 7f827e59c640 1 -- 172.21.15.63:0/3352170923 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8280104d40 con 0x7f8280075c40 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.437+0000 7f827e59c640 1 --2- 172.21.15.63:0/3352170923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8270009840 tx=0x7f82700091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43354fd0b5420f4e server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.437+0000 7f827dd9b640 1 -- 172.21.15.63:0/3352170923 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270032440 con 0x7f8280075c40 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.437+0000 7f827dd9b640 1 -- 172.21.15.63:0/3352170923 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8270032a50 con 0x7f8280075c40 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.437+0000 7f827dd9b640 1 -- 172.21.15.63:0/3352170923 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8270031aa0 con 0x7f8280075c40 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- 172.21.15.63:0/3352170923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 msgr2=0x7f82801046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 --2- 172.21.15.63:0/3352170923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8270009840 tx=0x7f82700091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- 172.21.15.63:0/3352170923 shutdown_connections 2024-06-15T00:01:36.216 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 --2- 172.21.15.63:0/3352170923 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f82801046c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- 172.21.15.63:0/3352170923 >> 172.21.15.63:0/3352170923 conn(0x7f82800fc150 msgr2=0x7f82800fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- 172.21.15.63:0/3352170923 shutdown_connections 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- 172.21.15.63:0/3352170923 wait complete. 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 Processor -- start 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.438+0000 7f8284943640 1 -- start start 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8284943640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8284943640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8280110dd0 con 0x7f8280075c40 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f827e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f827e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33150/0 (socket says 172.21.15.63:33150) 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f827e59c640 1 -- 172.21.15.63:0/2178790479 learned_addr learned my addr 172.21.15.63:0/2178790479 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f827e59c640 1 -- 172.21.15.63:0/2178790479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8270009450 con 0x7f8280075c40 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f827e59c640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8270031780 tx=0x7f827002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:36.217 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827003b440 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f827003ba50 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8280110fd0 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f827003aa70 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.439+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8280111440 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.440+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f8270039070 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.440+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828004b480 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.440+0000 7f8276ffd640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 0x7f826c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.440+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8270066ff0 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.441+0000 7f8277fff640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 0x7f826c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.441+0000 7f8277fff640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 0x7f826c036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8268006fd0 tx=0x7f8268006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.445+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f82700396e0 con 0x7f8280075c40 2024-06-15T00:01:36.218 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.620+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f8280111e60 con 0x7f8280075c40 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.620+0000 7f8276ffd640 1 -- 172.21.15.63:0/2178790479 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f827003f0c0 con 0x7f8280075c40 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 msgr2=0x7f826c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 0x7f826c036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8268006fd0 tx=0x7f8268006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 msgr2=0x7f8280110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8270031780 tx=0x7f827002c040 comp rx=0 tx=0).stop 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 shutdown_connections 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f826c034900 0x7f826c036db0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 --2- 172.21.15.63:0/2178790479 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f8280075c40 0x7f8280110890 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 >> 172.21.15.63:0/2178790479 conn(0x7f82800fc150 msgr2=0x7f82800fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 shutdown_connections 2024-06-15T00:01:36.219 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:35.622+0000 7f8284943640 1 -- 172.21.15.63:0/2178790479 wait complete. 2024-06-15T00:01:36.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:36 smithi063 ceph-mon[28882]: mgrmap e13: smithi063.dhizav(active, since 6s) 2024-06-15T00:01:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.608+0000 7fdf333d9640 1 Processor -- start 2024-06-15T00:01:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.608+0000 7fdf333d9640 1 -- start start 2024-06-15T00:01:37.437 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf333d9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf333d9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c078110 con 0x7fdf2c077d00 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf31175640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf31175640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33152/0 (socket says 172.21.15.63:33152) 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf31175640 1 -- 172.21.15.63:0/3091927759 learned_addr learned my addr 172.21.15.63:0/3091927759 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.609+0000 7fdf31175640 1 -- 172.21.15.63:0/3091927759 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf2c0767c0 con 0x7fdf2c077d00 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf31175640 1 --2- 172.21.15.63:0/3091927759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009840 tx=0x7fdf280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6be92af482633f1c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf23fff640 1 -- 172.21.15.63:0/3091927759 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28032440 con 0x7fdf2c077d00 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf23fff640 1 -- 172.21.15.63:0/3091927759 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf28032a50 con 0x7fdf2c077d00 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf23fff640 1 -- 172.21.15.63:0/3091927759 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf28031aa0 con 0x7fdf2c077d00 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf333d9640 1 -- 172.21.15.63:0/3091927759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 msgr2=0x7fdf2c076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf333d9640 1 --2- 172.21.15.63:0/3091927759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009840 tx=0x7fdf280091d0 comp rx=0 tx=0).stop 2024-06-15T00:01:37.438 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf333d9640 1 -- 172.21.15.63:0/3091927759 shutdown_connections 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf333d9640 1 --2- 172.21.15.63:0/3091927759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c076280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.610+0000 7fdf333d9640 1 -- 172.21.15.63:0/3091927759 >> 172.21.15.63:0/3091927759 conn(0x7fdf2c0fd280 msgr2=0x7fdf2c0ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 -- 172.21.15.63:0/3091927759 shutdown_connections 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 -- 172.21.15.63:0/3091927759 wait complete. 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 Processor -- start 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 -- start start 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf333d9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf2c112f80 con 0x7fdf2c077d00 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf31175640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf31175640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33164/0 (socket says 172.21.15.63:33164) 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.611+0000 7fdf31175640 1 -- 172.21.15.63:0/2644204151 learned_addr learned my addr 172.21.15.63:0/2644204151 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf31175640 1 -- 172.21.15.63:0/2644204151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf28009450 con 0x7fdf2c077d00 2024-06-15T00:01:37.439 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf31175640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdf28031780 tx=0x7fdf2802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2803b440 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf2803ba50 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf2c113180 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf2803aa70 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.612+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf2c1135f0 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.613+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fdf28039070 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.613+0000 7fdf21ffb640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 0x7fdf18036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.613+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdf28067170 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.613+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf2c05ead0 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.613+0000 7fdf30974640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 0x7fdf18036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.614+0000 7fdf30974640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 0x7fdf18036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c006fd0 tx=0x7fdf1c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.618+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdf28037070 con 0x7fdf2c077d00 2024-06-15T00:01:37.440 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.879+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fdf2c04b480 con 0x7fdf2c077d00 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.881+0000 7fdf21ffb640 1 -- 172.21.15.63:0/2644204151 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fdf2803b5a0 con 0x7fdf2c077d00 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 msgr2=0x7fdf18036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 0x7fdf18036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c006fd0 tx=0x7fdf1c006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 msgr2=0x7fdf2c112a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fdf28031780 tx=0x7fdf2802c040 comp rx=0 tx=0).stop 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 shutdown_connections 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fdf18034950 0x7fdf18036e00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 --2- 172.21.15.63:0/2644204151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fdf2c077d00 0x7fdf2c112a40 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 >> 172.21.15.63:0/2644204151 conn(0x7fdf2c0fd280 msgr2=0x7fdf2c0fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.883+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 shutdown_connections 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/bin/ceph: stderr 2024-06-15T00:01:36.884+0000 7fdf333d9640 1 -- 172.21.15.63:0/2644204151 wait complete. 2024-06-15T00:01:37.441 INFO:teuthology.orchestra.run.smithi063.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout:Or, if you are only running a single cluster on this host: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: ceph telemetry on 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout:For more information see: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.442 INFO:teuthology.orchestra.run.smithi063.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-15T00:01:37.443 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:37.443 INFO:teuthology.orchestra.run.smithi063.stdout:Bootstrap complete. 2024-06-15T00:01:37.476 INFO:tasks.cephadm:Fetching config... 2024-06-15T00:01:37.476 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:01:37.476 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-15T00:01:37.494 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-15T00:01:37.494 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:01:37.494 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-15T00:01:37.549 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-15T00:01:37.550 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:01:37.550 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/keyring of=/dev/stdout 2024-06-15T00:01:37.615 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-15T00:01:37.616 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:01:37.616 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-15T00:01:37.672 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-15T00:01:37.672 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fwGN8sRqUy0kcQ2MJZ6BTPzbJFgGzOlh9HP/XZBesrXLlKyJXgKcn2AD6HH8/Uc4fjd3uE9/IFNJpi7maERSThmeRcbQgmG/gQXmZvAh8BDl7aIZgpCOWKoVpM5xJNQRlStHs0QHymBFMu0+6kRasqNWSEcm+mLYmqkXnZB8afcZf2cgaY+MrKPakh5vxRyirJHOySYlBUB73Uo/mW8mfBpjjtf1jkIWGftfd9iXfeDGw74Y2oUeNuwEn9oHfbzx9zbEQIT1UYs2/e6mP/MbrZzoUvvLsptZAhOgmQCmTVBSnm0P8twvFpJ91SQSMt/rFxddD0mk6clHyMgX1sbIGRSk5TOnMzwj8amlHJJmKj0bA8bDR4CISkNrponfhun6ymRALoium5gT13rn5fLqqi5Ti/lQMH3hG5j3ZeyAn81hLYtr/ute2XwCrLiDKLo5RigMtDQBxQxi3sx/ZiCiHLQB5XRJeVVnQNrDOtfp4ng/KRx0XTHDcgAIDSjrFQE= ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-15T00:01:37.750 INFO:teuthology.orchestra.run.smithi063.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fwGN8sRqUy0kcQ2MJZ6BTPzbJFgGzOlh9HP/XZBesrXLlKyJXgKcn2AD6HH8/Uc4fjd3uE9/IFNJpi7maERSThmeRcbQgmG/gQXmZvAh8BDl7aIZgpCOWKoVpM5xJNQRlStHs0QHymBFMu0+6kRasqNWSEcm+mLYmqkXnZB8afcZf2cgaY+MrKPakh5vxRyirJHOySYlBUB73Uo/mW8mfBpjjtf1jkIWGftfd9iXfeDGw74Y2oUeNuwEn9oHfbzx9zbEQIT1UYs2/e6mP/MbrZzoUvvLsptZAhOgmQCmTVBSnm0P8twvFpJ91SQSMt/rFxddD0mk6clHyMgX1sbIGRSk5TOnMzwj8amlHJJmKj0bA8bDR4CISkNrponfhun6ymRALoium5gT13rn5fLqqi5Ti/lQMH3hG5j3ZeyAn81hLYtr/ute2XwCrLiDKLo5RigMtDQBxQxi3sx/ZiCiHLQB5XRJeVVnQNrDOtfp4ng/KRx0XTHDcgAIDSjrFQE= ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:01:37.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fwGN8sRqUy0kcQ2MJZ6BTPzbJFgGzOlh9HP/XZBesrXLlKyJXgKcn2AD6HH8/Uc4fjd3uE9/IFNJpi7maERSThmeRcbQgmG/gQXmZvAh8BDl7aIZgpCOWKoVpM5xJNQRlStHs0QHymBFMu0+6kRasqNWSEcm+mLYmqkXnZB8afcZf2cgaY+MrKPakh5vxRyirJHOySYlBUB73Uo/mW8mfBpjjtf1jkIWGftfd9iXfeDGw74Y2oUeNuwEn9oHfbzx9zbEQIT1UYs2/e6mP/MbrZzoUvvLsptZAhOgmQCmTVBSnm0P8twvFpJ91SQSMt/rFxddD0mk6clHyMgX1sbIGRSk5TOnMzwj8amlHJJmKj0bA8bDR4CISkNrponfhun6ymRALoium5gT13rn5fLqqi5Ti/lQMH3hG5j3ZeyAn81hLYtr/ute2XwCrLiDKLo5RigMtDQBxQxi3sx/ZiCiHLQB5XRJeVVnQNrDOtfp4ng/KRx0XTHDcgAIDSjrFQE= ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-15T00:01:37.806 INFO:teuthology.orchestra.run.smithi136.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2fwGN8sRqUy0kcQ2MJZ6BTPzbJFgGzOlh9HP/XZBesrXLlKyJXgKcn2AD6HH8/Uc4fjd3uE9/IFNJpi7maERSThmeRcbQgmG/gQXmZvAh8BDl7aIZgpCOWKoVpM5xJNQRlStHs0QHymBFMu0+6kRasqNWSEcm+mLYmqkXnZB8afcZf2cgaY+MrKPakh5vxRyirJHOySYlBUB73Uo/mW8mfBpjjtf1jkIWGftfd9iXfeDGw74Y2oUeNuwEn9oHfbzx9zbEQIT1UYs2/e6mP/MbrZzoUvvLsptZAhOgmQCmTVBSnm0P8twvFpJ91SQSMt/rFxddD0mk6clHyMgX1sbIGRSk5TOnMzwj8amlHJJmKj0bA8bDR4CISkNrponfhun6ymRALoium5gT13rn5fLqqi5Ti/lQMH3hG5j3ZeyAn81hLYtr/ute2XwCrLiDKLo5RigMtDQBxQxi3sx/ZiCiHLQB5XRJeVVnQNrDOtfp4ng/KRx0XTHDcgAIDSjrFQE= ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:01:37.819 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-15T00:01:38.067 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:37 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2644204151' entity='client.admin' 2024-06-15T00:01:38.139 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:01:40.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- 172.21.15.63:0/2226886959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 msgr2=0x7f56780fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 --2- 172.21.15.63:0/2226886959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f56780fc320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5668004850 tx=0x7f5668009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- 172.21.15.63:0/2226886959 shutdown_connections 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 --2- 172.21.15.63:0/2226886959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f56780fc320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- 172.21.15.63:0/2226886959 >> 172.21.15.63:0/2226886959 conn(0x7f56780f76f0 msgr2=0x7f56780f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- 172.21.15.63:0/2226886959 shutdown_connections 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- 172.21.15.63:0/2226886959 wait complete. 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 Processor -- start 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.667+0000 7f567cb5e640 1 -- start start 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567cb5e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567cb5e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567810c9b0 con 0x7f56780fbf10 2024-06-15T00:01:40.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57184/0 (socket says 172.21.15.63:57184) 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567659c640 1 -- 172.21.15.63:0/1842273601 learned_addr learned my addr 172.21.15.63:0/1842273601 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567659c640 1 -- 172.21.15.63:0/1842273601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668005040 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567659c640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5668009fa0 tx=0x7f5668009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5668033630 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567810cbb0 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5668033c40 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f566803cc60 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.668+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567810d020 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.669+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f5668030020 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.669+0000 7f566effd640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 0x7f5664036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.669+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f56680662a0 con 0x7f56780fbf10 2024-06-15T00:01:40.678 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.669+0000 7f5675d9b640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 0x7f5664036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:40.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.669+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5678102340 con 0x7f56780fbf10 2024-06-15T00:01:40.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.670+0000 7f5675d9b640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 0x7f5664036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:40.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.674+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f56680323d0 con 0x7f56780fbf10 2024-06-15T00:01:40.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.851+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f567805ead0 con 0x7f56780fbf10 2024-06-15T00:01:40.854 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.855+0000 7f566effd640 1 -- 172.21.15.63:0/1842273601 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f566804b0c0 con 0x7f56780fbf10 2024-06-15T00:01:40.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 msgr2=0x7f5664036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:40.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 0x7f5664036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 msgr2=0x7f567810c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5668009fa0 tx=0x7f5668009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 shutdown_connections 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f5664034900 0x7f5664036db0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 --2- 172.21.15.63:0/1842273601 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56780fbf10 0x7f567810c470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 >> 172.21.15.63:0/1842273601 conn(0x7f56780f76f0 msgr2=0x7f56780f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 shutdown_connections 2024-06-15T00:01:40.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:40.857+0000 7f567cb5e640 1 -- 172.21.15.63:0/1842273601 wait complete. 2024-06-15T00:01:41.664 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-15T00:01:41.664 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-15T00:01:41.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:41 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1842273601' entity='client.admin' 2024-06-15T00:01:42.160 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:01:43.351 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:43.351 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:43.351 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:01:43.351 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:43.352 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:01:43.352 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-15T00:01:43.352 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:01:43.352 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:43 smithi063 ceph-mon[28882]: Deploying daemon crash.smithi063 on smithi063 2024-06-15T00:01:43.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 -- 172.21.15.63:0/3569377610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 msgr2=0x7f80100fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:43.684 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 --2- 172.21.15.63:0/3569377610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f80100fb2e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f800c004850 tx=0x7f800c009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 -- 172.21.15.63:0/3569377610 shutdown_connections 2024-06-15T00:01:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 --2- 172.21.15.63:0/3569377610 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f80100fb2e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 -- 172.21.15.63:0/3569377610 >> 172.21.15.63:0/3569377610 conn(0x7f80100f65f0 msgr2=0x7f80100f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 -- 172.21.15.63:0/3569377610 shutdown_connections 2024-06-15T00:01:43.685 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.681+0000 7f801732d640 1 -- 172.21.15.63:0/3569377610 wait complete. 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.682+0000 7f801732d640 1 Processor -- start 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.682+0000 7f801732d640 1 -- start start 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f801732d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f801732d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801010eb80 con 0x7f80100faed0 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f80150c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:43.686 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f80150c9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57194/0 (socket says 172.21.15.63:57194) 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f80150c9640 1 -- 172.21.15.63:0/4284111932 learned_addr learned my addr 172.21.15.63:0/4284111932 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f80150c9640 1 -- 172.21.15.63:0/4284111932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f800c005040 con 0x7f80100faed0 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f80150c9640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f800c009fa0 tx=0x7f800c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f800c033630 con 0x7f80100faed0 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f800c033c40 con 0x7f80100faed0 2024-06-15T00:01:43.687 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801010ed80 con 0x7f80100faed0 2024-06-15T00:01:43.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.683+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f800c03cc60 con 0x7f80100faed0 2024-06-15T00:01:43.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.684+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801010f1f0 con 0x7f80100faed0 2024-06-15T00:01:43.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.684+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f800c030020 con 0x7f80100faed0 2024-06-15T00:01:43.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.685+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801005ead0 con 0x7f80100faed0 2024-06-15T00:01:43.688 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.685+0000 7f8005ffb640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 0x7f7ffc036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:43.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.685+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f800c066370 con 0x7f80100faed0 2024-06-15T00:01:43.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.685+0000 7f80148c8640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 0x7f7ffc036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:43.689 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.685+0000 7f80148c8640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 0x7f7ffc036e00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8000006fd0 tx=0x7f8000006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:43.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.689+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f800c03d990 con 0x7f80100faed0 2024-06-15T00:01:43.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.874+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f801010f1f0 con 0x7f7ffc034950 2024-06-15T00:01:43.876 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.878+0000 7f8005ffb640 1 -- 172.21.15.63:0/4284111932 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f801010f1f0 con 0x7f7ffc034950 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 msgr2=0x7f7ffc036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 0x7f7ffc036e00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8000006fd0 tx=0x7f8000006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 msgr2=0x7f801010e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f800c009fa0 tx=0x7f800c009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 shutdown_connections 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7ffc034950 0x7f7ffc036e00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 --2- 172.21.15.63:0/4284111932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f80100faed0 0x7f801010e640 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.879+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 >> 172.21.15.63:0/4284111932 conn(0x7f80100f65f0 msgr2=0x7f80100f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.880+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 shutdown_connections 2024-06-15T00:01:43.879 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:43.880+0000 7f801732d640 1 -- 172.21.15.63:0/4284111932 wait complete. 2024-06-15T00:01:44.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:44 smithi063 ceph-mon[28882]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:44.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:44 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:45.232 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi136 2024-06-15T00:01:45.233 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:01:45.233 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.conf 2024-06-15T00:01:45.252 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:01:45.252 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:01:45.309 INFO:tasks.cephadm:Adding host smithi136 to orchestrator... 2024-06-15T00:01:45.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch host add smithi136 2024-06-15T00:01:45.772 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: Deploying daemon node-exporter.smithi063 on smithi063 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.887 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:46.888 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:46 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 -- 172.21.15.63:0/3772708241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 msgr2=0x7f92500fa720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 --2- 172.21.15.63:0/3772708241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f92500fa720 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9240004850 tx=0x7f9240009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 -- 172.21.15.63:0/3772708241 shutdown_connections 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 --2- 172.21.15.63:0/3772708241 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f92500fa720 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 -- 172.21.15.63:0/3772708241 >> 172.21.15.63:0/3772708241 conn(0x7f92500f5d70 msgr2=0x7f92500f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:47.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 -- 172.21.15.63:0/3772708241 shutdown_connections 2024-06-15T00:01:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.364+0000 7f92552f0640 1 -- 172.21.15.63:0/3772708241 wait complete. 2024-06-15T00:01:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.365+0000 7f92552f0640 1 Processor -- start 2024-06-15T00:01:47.365 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.365+0000 7f92552f0640 1 -- start start 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f92552f0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f92552f0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925010cbe0 con 0x7f92500fa310 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f924effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f924effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:57262/0 (socket says 172.21.15.63:57262) 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f924effd640 1 -- 172.21.15.63:0/3649486997 learned_addr learned my addr 172.21.15.63:0/3649486997 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f924effd640 1 -- 172.21.15.63:0/3649486997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9240005040 con 0x7f92500fa310 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.366+0000 7f924effd640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9240009fa0 tx=0x7f9240009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:47.366 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.367+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9240033630 con 0x7f92500fa310 2024-06-15T00:01:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.367+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9240033c40 con 0x7f92500fa310 2024-06-15T00:01:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.367+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f924003cc60 con 0x7f92500fa310 2024-06-15T00:01:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.367+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f925010cde0 con 0x7f92500fa310 2024-06-15T00:01:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.367+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925010d250 con 0x7f92500fa310 2024-06-15T00:01:47.367 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f9240030020 con 0x7f92500fa310 2024-06-15T00:01:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f92377fe640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 0x7f9238036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925005ead0 con 0x7f92500fa310 2024-06-15T00:01:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9240066260 con 0x7f92500fa310 2024-06-15T00:01:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f924e7fc640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 0x7f9238036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:47.368 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.368+0000 7f924e7fc640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 0x7f9238036e00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9244006fd0 tx=0x7f9244006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:47.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.374+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f924003a2d0 con 0x7f92500fa310 2024-06-15T00:01:47.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:47.575+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}) v1 -- 0x7f9250101ef0 con 0x7f9238034950 2024-06-15T00:01:48.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:47 smithi063 ceph-mon[28882]: from='client.14186 -' entity='client.crash.smithi063' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:48.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:47 smithi063 ceph-mon[28882]: from='client.14188 -' entity='client.crash.smithi063' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:48.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:47 smithi063 ceph-mon[28882]: from='client.14190 -' entity='client.crash.smithi063' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:49.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:48 smithi063 ceph-mon[28882]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:01:49.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:48 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:50.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:49 smithi063 ceph-mon[28882]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:50.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:49 smithi063 ceph-mon[28882]: Deploying cephadm binary to smithi136 2024-06-15T00:01:50.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:49 smithi063 ceph-mon[28882]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-15T00:01:51.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.103+0000 7f92377fe640 1 -- 172.21.15.63:0/3649486997 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9250101ef0 con 0x7f9238034950 2024-06-15T00:01:51.103 INFO:teuthology.orchestra.run.smithi063.stdout:Added host 'smithi136' with addr '172.21.15.136' 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.105+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 msgr2=0x7f9238036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.105+0000 7f92552f0640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 0x7f9238036e00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9244006fd0 tx=0x7f9244006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.105+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 msgr2=0x7f925010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.105+0000 7f92552f0640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9240009fa0 tx=0x7f9240009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.105+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 shutdown_connections 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.106+0000 7f92552f0640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f9238034950 0x7f9238036e00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.106+0000 7f92552f0640 1 --2- 172.21.15.63:0/3649486997 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f92500fa310 0x7f925010c6a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.106+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 >> 172.21.15.63:0/3649486997 conn(0x7f92500f5d70 msgr2=0x7f92500f65c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:51.105 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.106+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 shutdown_connections 2024-06-15T00:01:51.106 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:51.106+0000 7f92552f0640 1 -- 172.21.15.63:0/3649486997 wait complete. 2024-06-15T00:01:51.755 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-15T00:01:51.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: Deploying daemon alertmanager.smithi063 on smithi063 2024-06-15T00:01:51.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:51 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:52.017 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:01:52.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:52 smithi063 ceph-mon[28882]: Added host smithi136 2024-06-15T00:01:53.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:53 smithi063 ceph-mon[28882]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:53.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:53 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:01:55.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- 172.21.15.63:0/548532389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 msgr2=0x7fbfb80fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:55.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/548532389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb80fc340 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004850 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- 172.21.15.63:0/548532389 shutdown_connections 2024-06-15T00:01:55.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/548532389 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb80fc340 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- 172.21.15.63:0/548532389 >> 172.21.15.63:0/548532389 conn(0x7fbfb80f7710 msgr2=0x7fbfb80f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- 172.21.15.63:0/548532389 shutdown_connections 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- 172.21.15.63:0/548532389 wait complete. 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 Processor -- start 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.114+0000 7fbfbce5a640 1 -- start start 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfbce5a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfbce5a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb810c9a0 con 0x7fbfb80fbf30 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfb659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfb659c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44654/0 (socket says 172.21.15.63:44654) 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfb659c640 1 -- 172.21.15.63:0/1554181655 learned_addr learned my addr 172.21.15.63:0/1554181655 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfb659c640 1 -- 172.21.15.63:0/1554181655 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005040 con 0x7fbfb80fbf30 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.115+0000 7fbfb659c640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009fa0 tx=0x7fbfa8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.116+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa8033630 con 0x7fbfb80fbf30 2024-06-15T00:01:55.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.116+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb810cba0 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.116+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfa8033c40 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.116+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa803cc60 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.116+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb810d010 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fbfa8030020 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb805ead0 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfaeffd640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbfa8066290 con 0x7fbfb80fbf30 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfb5d9b640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.117+0000 7fbfb5d9b640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006fd0 tx=0x7fbfa0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:01:55.185 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.122+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbfa803bb20 con 0x7fbfb80fbf30 2024-06-15T00:01:55.300 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.302+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbfb810d010 con 0x7fbfa4034900 2024-06-15T00:01:55.301 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.303+0000 7fbfaeffd640 1 -- 172.21.15.63:0/1554181655 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fbfb810d010 con 0x7fbfa4034900 2024-06-15T00:01:55.302 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:01:55.302 INFO:teuthology.orchestra.run.smithi063.stdout:[{"addr": "172.21.15.63", "hostname": "smithi063", "labels": [], "status": ""}, {"addr": "172.21.15.136", "hostname": "smithi136", "labels": [], "status": ""}] 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 msgr2=0x7fbfa4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbfa0006fd0 tx=0x7fbfa0006db0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 msgr2=0x7fbfb810c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8009fa0 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 shutdown_connections 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 --2- 172.21.15.63:0/1554181655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fbf30 0x7fbfb810c460 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 >> 172.21.15.63:0/1554181655 conn(0x7fbfb80f7710 msgr2=0x7fbfb80f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:01:55.304 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 shutdown_connections 2024-06-15T00:01:55.305 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:01:55.305+0000 7fbfbce5a640 1 -- 172.21.15.63:0/1554181655 wait complete. 2024-06-15T00:01:55.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:55 smithi063 ceph-mon[28882]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:56.087 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-15T00:01:56.087 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-15T00:01:56.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:56 smithi063 ceph-mon[28882]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:01:57.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:57 smithi063 ceph-mon[28882]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:01:59.168 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:01:59.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:01:59 smithi063 ceph-mon[28882]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 -- 172.21.15.63:0/3072949774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 msgr2=0x7f08180fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 --2- 172.21.15.63:0/3072949774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fec00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f080c004850 tx=0x7f080c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 -- 172.21.15.63:0/3072949774 shutdown_connections 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 --2- 172.21.15.63:0/3072949774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fec00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 -- 172.21.15.63:0/3072949774 >> 172.21.15.63:0/3072949774 conn(0x7f08180f65f0 msgr2=0x7f08180f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:00.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 -- 172.21.15.63:0/3072949774 shutdown_connections 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.891+0000 7f081f0ff640 1 -- 172.21.15.63:0/3072949774 wait complete. 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081f0ff640 1 Processor -- start 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081f0ff640 1 -- start start 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081f0ff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081f0ff640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08180fca70 con 0x7f0818065e40 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081ce9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081ce9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:33588/0 (socket says 172.21.15.63:33588) 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.892+0000 7f081ce9b640 1 -- 172.21.15.63:0/1681881900 learned_addr learned my addr 172.21.15.63:0/1681881900 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f081ce9b640 1 -- 172.21.15.63:0/1681881900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c005040 con 0x7f0818065e40 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f081ce9b640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f080c003080 tx=0x7f080c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f080c033630 con 0x7f0818065e40 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08180fab00 con 0x7f0818065e40 2024-06-15T00:02:00.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f080c033c40 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f080c03dc50 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.893+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08180faf70 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.894+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f080c030020 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.894+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0818102360 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.894+0000 7f08157fa640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 0x7f0808036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.894+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f080c0669b0 con 0x7f0818065e40 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.894+0000 7f0817fff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 0x7f0808036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.895+0000 7f0817fff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 0x7f0808036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0804006fd0 tx=0x7f0804006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:00.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:00.898+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f080c0323d0 con 0x7f0818065e40 2024-06-15T00:02:01.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.078+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f081805ead0 con 0x7f0818065e40 2024-06-15T00:02:01.668 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2024-06-15T00:02:01.668 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.669+0000 7f08157fa640 1 -- 172.21.15.63:0/1681881900 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f080c044340 con 0x7f0818065e40 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.671+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 msgr2=0x7f0808036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.671+0000 7f081f0ff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 0x7f0808036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0804006fd0 tx=0x7f0804006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.671+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 msgr2=0x7f08180fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.671+0000 7f081f0ff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f080c003080 tx=0x7f080c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 shutdown_connections 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0808034900 0x7f0808036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 --2- 172.21.15.63:0/1681881900 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0818065e40 0x7f08180fc530 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 >> 172.21.15.63:0/1681881900 conn(0x7f08180f65f0 msgr2=0x7f0818102f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 shutdown_connections 2024-06-15T00:02:01.671 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:02:01.672+0000 7f081f0ff640 1 -- 172.21.15.63:0/1681881900 wait complete. 2024-06-15T00:02:01.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:01.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1681881900' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:02:01.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:01 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:02.468 INFO:tasks.cephadm:Adding mon.smithi063 on smithi063 2024-06-15T00:02:02.468 INFO:tasks.cephadm:Adding mon.smithi136 on smithi136 2024-06-15T00:02:02.468 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch apply mon '2;smithi063:172.21.15.63=smithi063;smithi136:172.21.15.136=smithi136' 2024-06-15T00:02:02.716 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:02.764 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:02.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:02 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:02:02.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:02 smithi063 ceph-mon[28882]: Deploying daemon grafana.smithi063 on smithi063 2024-06-15T00:02:02.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:02 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1681881900' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-15T00:02:02.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:02 smithi063 ceph-mon[28882]: osdmap e4: 0 total, 0 up, 0 in 2024-06-15T00:02:03.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:03 smithi063 ceph-mon[28882]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:03.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:03 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:04.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 -- 172.21.15.136:0/84271665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 msgr2=0x7f35e80fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 --2- 172.21.15.136:0/84271665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e80fb300 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f35d80048d0 tx=0x7f35d8009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 -- 172.21.15.136:0/84271665 shutdown_connections 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 --2- 172.21.15.136:0/84271665 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e80fb300 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 -- 172.21.15.136:0/84271665 >> 172.21.15.136:0/84271665 conn(0x7f35e80f6610 msgr2=0x7f35e80f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.267+0000 7f35ed84e640 1 -- 172.21.15.136:0/84271665 shutdown_connections 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35ed84e640 1 -- 172.21.15.136:0/84271665 wait complete. 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35ed84e640 1 Processor -- start 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35ed84e640 1 -- start start 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35ed84e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:04.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35ed84e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e810a800 con 0x7f35e80faef0 2024-06-15T00:02:04.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.268+0000 7f35e6ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:04.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35e6ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:34078/0 (socket says 172.21.15.136:34078) 2024-06-15T00:02:04.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35e6ffd640 1 -- 172.21.15.136:0/3984469186 learned_addr learned my addr 172.21.15.136:0/3984469186 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:04.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35e6ffd640 1 -- 172.21.15.136:0/3984469186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d8005040 con 0x7f35e80faef0 2024-06-15T00:02:04.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35e6ffd640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35d80048a0 tx=0x7f35d8038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8034c80 con 0x7f35e80faef0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35d8034de0 con 0x7f35e80faef0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35d8040a40 con 0x7f35e80faef0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e810aa00 con 0x7f35e80faef0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.269+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e810ae70 con 0x7f35e80faef0 2024-06-15T00:02:04.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.270+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f35d8040c30 con 0x7f35e80faef0 2024-06-15T00:02:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.270+0000 7f35dffff640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 0x7f35d4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.270+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f35d80684a0 con 0x7f35e80faef0 2024-06-15T00:02:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.270+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e8100170 con 0x7f35e80faef0 2024-06-15T00:02:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.270+0000 7f35de7fc640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 0x7f35d4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.271+0000 7f35de7fc640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 0x7f35d4036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:04.273 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.275+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f35d8033dd0 con 0x7f35e80faef0 2024-06-15T00:02:04.447 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.448+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 --> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi063:172.21.15.63=smithi063;smithi136:172.21.15.136=smithi136", "target": ["mon-mgr", ""]}) v1 -- 0x7f35e810b5f0 con 0x7f35d4034900 2024-06-15T00:02:04.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.453+0000 7f35dffff640 1 -- 172.21.15.136:0/3984469186 <== mgr.14160 v2:172.21.15.63:6800/3768764057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f35e810b5f0 con 0x7f35d4034900 2024-06-15T00:02:04.452 INFO:teuthology.orchestra.run.smithi136.stdout:Scheduled mon update... 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 msgr2=0x7f35d4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 0x7f35d4036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f35d0006fd0 tx=0x7f35d0006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 msgr2=0x7f35e810a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f35d80048a0 tx=0x7f35d8038040 comp rx=0 tx=0).stop 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 shutdown_connections 2024-06-15T00:02:04.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f35d4034900 0x7f35d4036db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:04.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 --2- 172.21.15.136:0/3984469186 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f35e80faef0 0x7f35e810a2c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:04.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 >> 172.21.15.136:0/3984469186 conn(0x7f35e80f6610 msgr2=0x7f35e8100d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:04.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 shutdown_connections 2024-06-15T00:02:04.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:04.455+0000 7f35ed84e640 1 -- 172.21.15.136:0/3984469186 wait complete. 2024-06-15T00:02:05.034 DEBUG:teuthology.orchestra.run.smithi136:mon.smithi136> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi136.service 2024-06-15T00:02:05.037 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:05.037 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:05.311 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:05.359 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:05.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:05 smithi063 ceph-mon[28882]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi063:172.21.15.63=smithi063;smithi136:172.21.15.136=smithi136", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:02:05.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:05 smithi063 ceph-mon[28882]: Saving service mon spec with placement smithi063:172.21.15.63=smithi063;smithi136:172.21.15.136=smithi136;count:2 2024-06-15T00:02:05.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:05 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:05.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:05 smithi063 ceph-mon[28882]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:06.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.803+0000 7f4fb5565640 1 -- 172.21.15.136:0/3478237116 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fac032590 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- 172.21.15.136:0/3478237116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 msgr2=0x7f4fb00fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/3478237116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb00fd9f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4fac004850 tx=0x7f4fac009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- 172.21.15.136:0/3478237116 shutdown_connections 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/3478237116 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb00fd9f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- 172.21.15.136:0/3478237116 >> 172.21.15.136:0/3478237116 conn(0x7f4fb00f5da0 msgr2=0x7f4fb00f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- 172.21.15.136:0/3478237116 shutdown_connections 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- 172.21.15.136:0/3478237116 wait complete. 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 Processor -- start 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.804+0000 7f4fb87cb640 1 -- start start 2024-06-15T00:02:06.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb87cb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb87cb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb010c980 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb6567640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb6567640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:34096/0 (socket says 172.21.15.136:34096) 2024-06-15T00:02:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb6567640 1 -- 172.21.15.136:0/369949652 learned_addr learned my addr 172.21.15.136:0/369949652 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:06.805 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb6567640 1 -- 172.21.15.136:0/369949652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fac005040 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb6567640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009fa0 tx=0x7f4fac009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:06.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fac033630 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fac033c40 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.805+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb010cb80 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.806 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.806+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fac03cc60 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.806+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb010cff0 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.806+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f4fac030020 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.806+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb005ead0 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.806+0000 7f4fa6ffd640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 0x7f4f9c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.807+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4fac066410 con 0x7f4fb00fd5e0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.807+0000 7f4fb5d66640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 0x7f4f9c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:06.807 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.807+0000 7f4fb5d66640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 0x7f4f9c036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0006fd0 tx=0x7f4fa0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:06.810 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:06.811+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4fac043940 con 0x7f4fb00fd5e0 2024-06-15T00:02:07.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.036+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fb004b480 con 0x7f4fb00fd5e0 2024-06-15T00:02:07.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.037+0000 7f4fa6ffd640 1 -- 172.21.15.136:0/369949652 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f4fac03c3d0 con 0x7f4fb00fd5e0 2024-06-15T00:02:07.035 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:07.036 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:07.036 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:07.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 msgr2=0x7f4f9c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 0x7f4f9c036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4fa0006fd0 tx=0x7f4fa0006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 msgr2=0x7f4fb010c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4fac009fa0 tx=0x7f4fac009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 shutdown_connections 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f4f9c034900 0x7f4f9c036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 --2- 172.21.15.136:0/369949652 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4fb00fd5e0 0x7f4fb010c440 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 >> 172.21.15.136:0/369949652 conn(0x7f4fb00f5da0 msgr2=0x7f4fb00f65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 shutdown_connections 2024-06-15T00:02:07.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:07.038+0000 7f4fb87cb640 1 -- 172.21.15.136:0/369949652 wait complete. 2024-06-15T00:02:08.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:07 smithi063 ceph-mon[28882]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:08.044 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:07 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/369949652' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:08.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:08.587 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:08.832 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:08.881 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:10.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:09 smithi063 ceph-mon[28882]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:10.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 -- 172.21.15.136:0/3355603831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 msgr2=0x7fbfb80fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:10.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 --2- 172.21.15.136:0/3355603831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb80fb280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004850 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 -- 172.21.15.136:0/3355603831 shutdown_connections 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 --2- 172.21.15.136:0/3355603831 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb80fb280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 -- 172.21.15.136:0/3355603831 >> 172.21.15.136:0/3355603831 conn(0x7fbfb80f65e0 msgr2=0x7fbfb80f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 -- 172.21.15.136:0/3355603831 shutdown_connections 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.360+0000 7fbfc0737640 1 -- 172.21.15.136:0/3355603831 wait complete. 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfc0737640 1 Processor -- start 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfc0737640 1 -- start start 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfc0737640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfc0737640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfb810b930 con 0x7fbfb80fae70 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfbe4d3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfbe4d3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:34124/0 (socket says 172.21.15.136:34124) 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfbe4d3640 1 -- 172.21.15.136:0/2894831364 learned_addr learned my addr 172.21.15.136:0/2894831364 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:10.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfbe4d3640 1 -- 172.21.15.136:0/2894831364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa8005040 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.361+0000 7fbfbe4d3640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004820 tx=0x7fbfa8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.362+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa8007a20 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.362+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbfa8033ae0 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.362+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfb810bb30 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.362+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfa803db70 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.362+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfb810bfa0 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fbfa8030020 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfb805ead0 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfaeffd640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbfa8066ca0 con 0x7fbfb80fae70 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfbdcd2640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.363+0000 7fbfbdcd2640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4006fd0 tx=0x7fbfb4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:10.381 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.371+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbfa8033630 con 0x7fbfb80fae70 2024-06-15T00:02:10.597 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.599+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbfb804b480 con 0x7fbfb80fae70 2024-06-15T00:02:10.598 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.599+0000 7fbfaeffd640 1 -- 172.21.15.136:0/2894831364 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbfa8045390 con 0x7fbfb80fae70 2024-06-15T00:02:10.598 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:10.599 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:10.599 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 msgr2=0x7fbfa4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbfb4006fd0 tx=0x7fbfb4006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 msgr2=0x7fbfb810b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:10.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbfa8004820 tx=0x7fbfa8009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 shutdown_connections 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbfa4034900 0x7fbfa4036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 --2- 172.21.15.136:0/2894831364 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbfb80fae70 0x7fbfb810b3f0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.601+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 >> 172.21.15.136:0/2894831364 conn(0x7fbfb80f65e0 msgr2=0x7fbfb80f6ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.602+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 shutdown_connections 2024-06-15T00:02:10.601 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:10.602+0000 7fbfc0737640 1 -- 172.21.15.136:0/2894831364 wait complete. 2024-06-15T00:02:11.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:10 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2894831364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:11.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:11 smithi063 ceph-mon[28882]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:12.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:12.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:12.493 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:12.541 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:13.729 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:13 smithi063 ceph-mon[28882]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.945+0000 7fbd4e956640 1 -- 172.21.15.136:0/961420061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 msgr2=0x7fbd480fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.945+0000 7fbd4e956640 1 --2- 172.21.15.136:0/961420061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd480fb2e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fbd38004850 tx=0x7fbd38009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- 172.21.15.136:0/961420061 shutdown_connections 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 --2- 172.21.15.136:0/961420061 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd480fb2e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- 172.21.15.136:0/961420061 >> 172.21.15.136:0/961420061 conn(0x7fbd480f65f0 msgr2=0x7fbd480f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:13.948 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- 172.21.15.136:0/961420061 shutdown_connections 2024-06-15T00:02:13.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- 172.21.15.136:0/961420061 wait complete. 2024-06-15T00:02:13.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 Processor -- start 2024-06-15T00:02:13.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- start start 2024-06-15T00:02:13.949 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.946+0000 7fbd4e956640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd4810b870 con 0x7fbd480faed0 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd47fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:56822/0 (socket says 172.21.15.136:56822) 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd47fff640 1 -- 172.21.15.136:0/3007809156 learned_addr learned my addr 172.21.15.136:0/3007809156 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd47fff640 1 -- 172.21.15.136:0/3007809156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd38005040 con 0x7fbd480faed0 2024-06-15T00:02:13.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd47fff640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009fa0 tx=0x7fbd38009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:13.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd38033630 con 0x7fbd480faed0 2024-06-15T00:02:13.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbd38033c40 con 0x7fbd480faed0 2024-06-15T00:02:13.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4810ba70 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd3803cc60 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.947+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4810bee0 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.948+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fbd38030020 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.948+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4805ead0 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.948+0000 7fbd44ff9640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 0x7fbd30036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.948+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbd38066350 con 0x7fbd480faed0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.949+0000 7fbd477fe640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 0x7fbd30036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:13.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.949+0000 7fbd477fe640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 0x7fbd30036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c006fd0 tx=0x7fbd3c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:13.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:13.953+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbd3803bb20 con 0x7fbd480faed0 2024-06-15T00:02:14.176 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.177+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd4804b480 con 0x7fbd480faed0 2024-06-15T00:02:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.178+0000 7fbd44ff9640 1 -- 172.21.15.136:0/3007809156 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fbd3803bb20 con 0x7fbd480faed0 2024-06-15T00:02:14.177 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:14.177 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 msgr2=0x7fbd30036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 0x7fbd30036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c006fd0 tx=0x7fbd3c006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 msgr2=0x7fbd4810b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fbd38009fa0 tx=0x7fbd38009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 shutdown_connections 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fbd30034900 0x7fbd30036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 --2- 172.21.15.136:0/3007809156 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbd480faed0 0x7fbd4810b330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 >> 172.21.15.136:0/3007809156 conn(0x7fbd480f65f0 msgr2=0x7fbd48101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:14.179 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 shutdown_connections 2024-06-15T00:02:14.180 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:14.180+0000 7fbd4e956640 1 -- 172.21.15.136:0/3007809156 wait complete. 2024-06-15T00:02:14.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:14 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/3007809156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:15.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:15.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.809 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:15 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:15.961 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:16.009 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:16.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:16 smithi063 ceph-mon[28882]: Deploying daemon prometheus.smithi063 on smithi063 2024-06-15T00:02:17.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.349+0000 7f0f76026640 1 -- 172.21.15.136:0/2231463999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 msgr2=0x7f0f700fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:17.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.349+0000 7f0f76026640 1 --2- 172.21.15.136:0/2231463999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f700fb450 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0f60004850 tx=0x7f0f60009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- 172.21.15.136:0/2231463999 shutdown_connections 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 --2- 172.21.15.136:0/2231463999 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f700fb450 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- 172.21.15.136:0/2231463999 >> 172.21.15.136:0/2231463999 conn(0x7f0f700f6820 msgr2=0x7f0f700f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- 172.21.15.136:0/2231463999 shutdown_connections 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- 172.21.15.136:0/2231463999 wait complete. 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 Processor -- start 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- start start 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f76026640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7010d1a0 con 0x7f0f700fb040 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f6f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f6f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:56830/0 (socket says 172.21.15.136:56830) 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.350+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/2449061177 learned_addr learned my addr 172.21.15.136:0/2449061177 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:17.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f6f7fe640 1 -- 172.21.15.136:0/2449061177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f60005040 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f6f7fe640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009fa0 tx=0x7f0f60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f60033630 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f60033c40 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f701117b0 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6003cc60 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.351+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f70111ae0 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.352+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f0f60030020 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.352+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7005ead0 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.353+0000 7f0f6cff9640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 0x7f0f5c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.353+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0f600664f0 con 0x7f0f700fb040 2024-06-15T00:02:17.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.353+0000 7f0f67fff640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 0x7f0f5c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:17.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.353+0000 7f0f67fff640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 0x7f0f5c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0f58006fd0 tx=0x7f0f58006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:17.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.358+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0f6003bb70 con 0x7f0f700fb040 2024-06-15T00:02:17.581 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.583+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f7004b480 con 0x7f0f700fb040 2024-06-15T00:02:17.582 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.583+0000 7f0f6cff9640 1 -- 172.21.15.136:0/2449061177 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f0f60044340 con 0x7f0f700fb040 2024-06-15T00:02:17.582 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:17.582 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:17.583 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 msgr2=0x7f0f5c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 0x7f0f5c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0f58006fd0 tx=0x7f0f58006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 msgr2=0x7f0f70065230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0f60009fa0 tx=0x7f0f60009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 shutdown_connections 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0f5c034900 0x7f0f5c036db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 --2- 172.21.15.136:0/2449061177 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0f700fb040 0x7f0f70065230 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 >> 172.21.15.136:0/2449061177 conn(0x7f0f700f6820 msgr2=0x7f0f700f7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 shutdown_connections 2024-06-15T00:02:17.584 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:17.585+0000 7f0f76026640 1 -- 172.21.15.136:0/2449061177 wait complete. 2024-06-15T00:02:18.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:17 smithi063 ceph-mon[28882]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:18.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:17 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2449061177' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:19.111 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:19.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:19.360 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:19.407 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:19.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:19 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:19.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:19 smithi063 ceph-mon[28882]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:20.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/4258968930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 msgr2=0x7fe5cc0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:20.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/4258968930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc0fc320 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc004850 tx=0x7fe5bc009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/4258968930 shutdown_connections 2024-06-15T00:02:20.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/4258968930 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc0fc320 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/4258968930 >> 172.21.15.136:0/4258968930 conn(0x7fe5cc0f76f0 msgr2=0x7fe5cc0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:20.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/4258968930 shutdown_connections 2024-06-15T00:02:20.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.719+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/4258968930 wait complete. 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d3c3b640 1 Processor -- start 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d3c3b640 1 -- start start 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d3c3b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d3c3b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5cc10ebb0 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d19d7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d19d7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:56840/0 (socket says 172.21.15.136:56840) 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d19d7640 1 -- 172.21.15.136:0/509396195 learned_addr learned my addr 172.21.15.136:0/509396195 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d19d7640 1 -- 172.21.15.136:0/509396195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5bc005040 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.720+0000 7fe5d19d7640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009fa0 tx=0x7fe5bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:20.721 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5bc033630 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe5bc033c40 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5cc10edb0 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5bc03cc60 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5cc10f220 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.721+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7fe5bc030020 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.722+0000 7fe5c27fc640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 0x7fe5b8036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.722+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5cc05ead0 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.722+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fe5bc066360 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.722+0000 7fe5d11d6640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 0x7fe5b8036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:20.722 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.722+0000 7fe5d11d6640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 0x7fe5b8036e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:20.725 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.727+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe5bc03c3d0 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.954+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe5cc04b480 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.955+0000 7fe5c27fc640 1 -- 172.21.15.136:0/509396195 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe5bc0432f0 con 0x7fe5cc0fbf10 2024-06-15T00:02:20.954 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:20.954 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:20.955 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:20.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 msgr2=0x7fe5b8036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:20.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 0x7fe5b8036e00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8006fd0 tx=0x7fe5c8006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 msgr2=0x7fe5cc10e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:20.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fe5bc009fa0 tx=0x7fe5bc009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.956 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 shutdown_connections 2024-06-15T00:02:20.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7fe5b8034950 0x7fe5b8036e00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 --2- 172.21.15.136:0/509396195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe5cc0fbf10 0x7fe5cc10e670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:20.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 >> 172.21.15.136:0/509396195 conn(0x7fe5cc0f76f0 msgr2=0x7fe5cc0f9ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:20.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 shutdown_connections 2024-06-15T00:02:20.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:20.957+0000 7fe5d3c3b640 1 -- 172.21.15.136:0/509396195 wait complete. 2024-06-15T00:02:22.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:21 smithi063 ceph-mon[28882]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:22.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:21 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/509396195' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:22.581 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:22.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:22.830 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:22.878 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:23.805 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:23 smithi063 ceph-mon[28882]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:24.301 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- 172.21.15.136:0/1021328932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 msgr2=0x7f7c600fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/1021328932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c600fc340 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7c54004850 tx=0x7f7c54009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- 172.21.15.136:0/1021328932 shutdown_connections 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/1021328932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c600fc340 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- 172.21.15.136:0/1021328932 >> 172.21.15.136:0/1021328932 conn(0x7f7c600f7710 msgr2=0x7f7c600f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- 172.21.15.136:0/1021328932 shutdown_connections 2024-06-15T00:02:24.302 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- 172.21.15.136:0/1021328932 wait complete. 2024-06-15T00:02:24.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 Processor -- start 2024-06-15T00:02:24.303 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.303+0000 7f7c66dc9640 1 -- start start 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c66dc9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c66dc9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6010c990 con 0x7f7c600fbf30 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c64b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c64b65640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:49706/0 (socket says 172.21.15.136:49706) 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c64b65640 1 -- 172.21.15.136:0/756823277 learned_addr learned my addr 172.21.15.136:0/756823277 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.304+0000 7f7c64b65640 1 -- 172.21.15.136:0/756823277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54005040 con 0x7f7c600fbf30 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c64b65640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009fa0 tx=0x7f7c54009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c54033630 con 0x7f7c600fbf30 2024-06-15T00:02:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6010cb90 con 0x7f7c600fbf30 2024-06-15T00:02:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c54033c40 con 0x7f7c600fbf30 2024-06-15T00:02:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c5403cc60 con 0x7f7c600fbf30 2024-06-15T00:02:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.305+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6010d000 con 0x7f7c600fbf30 2024-06-15T00:02:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.306+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 13) v1 ==== 42002+0+0 (secure 0 0 0) 0x7f7c54030020 con 0x7f7c600fbf30 2024-06-15T00:02:24.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.306+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6005ead0 con 0x7f7c600fbf30 2024-06-15T00:02:24.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.306+0000 7f7c5d7fa640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 0x7f7c50036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:24.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.306+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7c54066370 con 0x7f7c600fbf30 2024-06-15T00:02:24.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.306+0000 7f7c5ffff640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 0x7f7c50036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:24.306 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.307+0000 7f7c5ffff640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 0x7f7c50036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c006fd0 tx=0x7f7c4c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:24.309 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.311+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c540432f0 con 0x7f7c600fbf30 2024-06-15T00:02:24.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.536+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c6004b480 con 0x7f7c600fbf30 2024-06-15T00:02:24.535 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.537+0000 7f7c5d7fa640 1 -- 172.21.15.136:0/756823277 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f7c540432f0 con 0x7f7c600fbf30 2024-06-15T00:02:24.536 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:24.536 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:24.536 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:24.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 msgr2=0x7f7c50036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 0x7f7c50036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c006fd0 tx=0x7f7c4c006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 msgr2=0x7f7c6010c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7c54009fa0 tx=0x7f7c54009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 shutdown_connections 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f7c50034900 0x7f7c50036db0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 --2- 172.21.15.136:0/756823277 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c600fbf30 0x7f7c6010c450 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 >> 172.21.15.136:0/756823277 conn(0x7f7c600f7710 msgr2=0x7f7c600f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 shutdown_connections 2024-06-15T00:02:24.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:24.539+0000 7f7c66dc9640 1 -- 172.21.15.136:0/756823277 wait complete. 2024-06-15T00:02:25.021 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/756823277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:25.975 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:25 smithi063 ceph-mon[28882]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:26.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:26.164 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:26.410 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:26.458 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:27.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:27 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:27.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:27 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:27.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:27 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:27.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:27 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-15T00:02:27.410 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:27 smithi063 ceph-mon[28882]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 -- 172.21.15.136:0/1278802713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 msgr2=0x7efe380feb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 --2- 172.21.15.136:0/1278802713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe380feb60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7efe30009bd0 tx=0x7efe30004d10 comp rx=0 tx=0).stop 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 -- 172.21.15.136:0/1278802713 shutdown_connections 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 --2- 172.21.15.136:0/1278802713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe380feb60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 -- 172.21.15.136:0/1278802713 >> 172.21.15.136:0/1278802713 conn(0x7efe380f6610 msgr2=0x7efe380f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 -- 172.21.15.136:0/1278802713 shutdown_connections 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.859+0000 7efe3d529640 1 -- 172.21.15.136:0/1278802713 wait complete. 2024-06-15T00:02:27.863 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe3d529640 1 Processor -- start 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe3d529640 1 -- start start 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe3d529640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe3d529640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3810c9f0 con 0x7efe38065ea0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe36ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe36ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:49724/0 (socket says 172.21.15.136:49724) 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe36ffd640 1 -- 172.21.15.136:0/2629899293 learned_addr learned my addr 172.21.15.136:0/2629899293 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.860+0000 7efe36ffd640 1 -- 172.21.15.136:0/2629899293 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300098b0 con 0x7efe38065ea0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe36ffd640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efe30003400 tx=0x7efe30004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe300328a0 con 0x7efe38065ea0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3810cbf0 con 0x7efe38065ea0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe3003d420 con 0x7efe38065ea0 2024-06-15T00:02:27.864 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe30031590 con 0x7efe38065ea0 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.861+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe3810d780 con 0x7efe38065ea0 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 14) v1 ==== 42016+0+0 (secure 0 0 0) 0x7efe3002f030 con 0x7efe38065ea0 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe2f7fe640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 0x7efe24036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe38102380 con 0x7efe38065ea0 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe367fc640 1 -- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 msgr2=0x7efe24036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3768764057 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7efe30067c60 con 0x7efe38065ea0 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.862+0000 7efe367fc640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 0x7efe24036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:02:27.865 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:27.867+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efe30030b70 con 0x7efe38065ea0 2024-06-15T00:02:28.061 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.063+0000 7efe367fc640 1 -- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 msgr2=0x7efe24036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3768764057 2024-06-15T00:02:28.062 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.063+0000 7efe367fc640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 0x7efe24036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:02:28.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.093+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe3810e0d0 con 0x7efe38065ea0 2024-06-15T00:02:28.092 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.093+0000 7efe2f7fe640 1 -- 172.21.15.136:0/2629899293 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7efe30030b70 con 0x7efe38065ea0 2024-06-15T00:02:28.092 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:28.093 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:28.093 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:28.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 msgr2=0x7efe24036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:02:28.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 0x7efe24036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:28.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 msgr2=0x7efe3810c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:28.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7efe30003400 tx=0x7efe30004d10 comp rx=0 tx=0).stop 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 shutdown_connections 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7efe24034950 0x7efe24036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 --2- 172.21.15.136:0/2629899293 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7efe38065ea0 0x7efe3810c4b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 >> 172.21.15.136:0/2629899293 conn(0x7efe380f6610 msgr2=0x7efe380f6f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 shutdown_connections 2024-06-15T00:02:28.095 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:28.095+0000 7efe3d529640 1 -- 172.21.15.136:0/2629899293 wait complete. 2024-06-15T00:02:28.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:28 smithi063 ceph-mon[28882]: from='mgr.14160 172.21.15.63:0/2123736804' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-15T00:02:28.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:28 smithi063 ceph-mon[28882]: mgrmap e14: smithi063.dhizav(active, since 58s) 2024-06-15T00:02:28.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:28 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2629899293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:29.666 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:29.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:29.918 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:29.966 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:31.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 -- 172.21.15.136:0/2803330440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 msgr2=0x7f07a80fc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2803330440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a80fc840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0798004850 tx=0x7f0798009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 -- 172.21.15.136:0/2803330440 shutdown_connections 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2803330440 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a80fc840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 -- 172.21.15.136:0/2803330440 >> 172.21.15.136:0/2803330440 conn(0x7f07a80f5d70 msgr2=0x7f07a80f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 -- 172.21.15.136:0/2803330440 shutdown_connections 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.369+0000 7f07adc0d640 1 -- 172.21.15.136:0/2803330440 wait complete. 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07adc0d640 1 Processor -- start 2024-06-15T00:02:31.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07adc0d640 1 -- start start 2024-06-15T00:02:31.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07adc0d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:31.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07adc0d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a810b8e0 con 0x7f07a80fc430 2024-06-15T00:02:31.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07a77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:31.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07a77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:49750/0 (socket says 172.21.15.136:49750) 2024-06-15T00:02:31.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.370+0000 7f07a77fe640 1 -- 172.21.15.136:0/2246497697 learned_addr learned my addr 172.21.15.136:0/2246497697 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f07a77fe640 1 -- 172.21.15.136:0/2246497697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0798005040 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f07a77fe640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0798009fa0 tx=0x7f0798009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0798033630 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a810bae0 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0798033c40 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f079803dd20 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.371+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a810bf50 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 14) v1 ==== 42016+0+0 (secure 0 0 0) 0x7f0798030020 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a805ead0 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f079ffff640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 0x7f0794036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0798066640 con 0x7f07a80fc430 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f07a6ffd640 1 -- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 msgr2=0x7f0794036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3768764057 2024-06-15T00:02:31.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.372+0000 7f07a6ffd640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 0x7f0794036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:02:31.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.377+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f079803b370 con 0x7f07a80fc430 2024-06-15T00:02:31.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.573+0000 7f07a6ffd640 1 -- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 msgr2=0x7f0794036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.63:6800/3768764057 2024-06-15T00:02:31.571 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.573+0000 7f07a6ffd640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 0x7f0794036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:02:31.599 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.601+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f07a804b480 con 0x7f07a80fc430 2024-06-15T00:02:31.600 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.602+0000 7f079ffff640 1 -- 172.21.15.136:0/2246497697 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f079803b370 con 0x7f07a80fc430 2024-06-15T00:02:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:31.601 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:31.601 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:31.602 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 msgr2=0x7f0794036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 0x7f0794036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 msgr2=0x7f07a810b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f0798009fa0 tx=0x7f0798009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 shutdown_connections 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:6800/3768764057,v1:172.21.15.63:6801/3768764057] conn(0x7f0794034950 0x7f0794036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 --2- 172.21.15.136:0/2246497697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f07a80fc430 0x7f07a810b3a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 >> 172.21.15.136:0/2246497697 conn(0x7f07a80f5d70 msgr2=0x7f07a80f6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 shutdown_connections 2024-06-15T00:02:31.603 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:31.604+0000 7f07adc0d640 1 -- 172.21.15.136:0/2246497697 wait complete. 2024-06-15T00:02:31.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:31 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2246497697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:32.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: Active manager daemon smithi063.dhizav restarted 2024-06-15T00:02:32.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: Activating manager daemon smithi063.dhizav 2024-06-15T00:02:32.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:02:32.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: mgrmap e15: smithi063.dhizav(active, starting, since 0.00289483s) 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr metadata", "who": "smithi063.dhizav", "id": "smithi063.dhizav"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: Manager daemon smithi063.dhizav is now available 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:02:32.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:32.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:32.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:02:32.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:02:32.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi063.dhizav/trash_purge_schedule"}]: dispatch 2024-06-15T00:02:32.932 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:33.231 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:33.231 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:33.479 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:33.528 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:34.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:33 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:34.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:33 smithi063 ceph-mon[28882]: mgrmap e16: smithi063.dhizav(active, since 1.00532s) 2024-06-15T00:02:34.041 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:33 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:35.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- 172.21.15.136:0/3299879942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 msgr2=0x7fe70c0fb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:35.004 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 --2- 172.21.15.136:0/3299879942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c0fb270 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc004850 tx=0x7fe6fc009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- 172.21.15.136:0/3299879942 shutdown_connections 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 --2- 172.21.15.136:0/3299879942 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c0fb270 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- 172.21.15.136:0/3299879942 >> 172.21.15.136:0/3299879942 conn(0x7fe70c0f65b0 msgr2=0x7fe70c0f8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- 172.21.15.136:0/3299879942 shutdown_connections 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- 172.21.15.136:0/3299879942 wait complete. 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 Processor -- start 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.005+0000 7fe710c78640 1 -- start start 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe710c78640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe710c78640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70c10c9f0 con 0x7fe70c0fae60 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe70a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:51562/0 (socket says 172.21.15.136:51562) 2024-06-15T00:02:35.005 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe70a59c640 1 -- 172.21.15.136:0/432423853 learned_addr learned my addr 172.21.15.136:0/432423853 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:35.006 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe70a59c640 1 -- 172.21.15.136:0/432423853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc005040 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe70a59c640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc003080 tx=0x7fe6fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc033630 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe70c10cbf0 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6fc033c40 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.006+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6fc03dc50 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.007+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70c10d060 con 0x7fe70c0fae60 2024-06-15T00:02:35.007 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fe6fc030020 con 0x7fe70c0fae60 2024-06-15T00:02:35.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe70c05ead0 con 0x7fe70c0fae60 2024-06-15T00:02:35.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe6faffd640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 0x7fe6f4036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:35.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fe6fc066930 con 0x7fe70c0fae60 2024-06-15T00:02:35.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe709d9b640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 0x7fe6f4036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:35.008 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.008+0000 7fe709d9b640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 0x7fe6f4036ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe700006fd0 tx=0x7fe700006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:35.014 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.016+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe6fc0323d0 con 0x7fe70c0fae60 2024-06-15T00:02:35.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: [15/Jun/2024:00:02:33] ENGINE Bus STARTING 2024-06-15T00:02:35.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: [15/Jun/2024:00:02:33] ENGINE Serving on https://172.21.15.63:7150 2024-06-15T00:02:35.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: [15/Jun/2024:00:02:33] ENGINE Bus STARTED 2024-06-15T00:02:35.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: [15/Jun/2024:00:02:33] ENGINE Error in HTTPServer.serve 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: Traceback (most recent call last): 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._run(expiration_interval) 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: s = self.context.wrap_socket( 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:02:35.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: return self.sslsocket_class._create( 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self.do_handshake() 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: self._sslobj.do_handshake() 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:35.181 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:34 smithi063 ceph-mon[28882]: mgrmap e17: smithi063.dhizav(active, since 2s) 2024-06-15T00:02:35.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.253+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe70c04b480 con 0x7fe70c0fae60 2024-06-15T00:02:35.252 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.254+0000 7fe6faffd640 1 -- 172.21.15.136:0/432423853 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fe6fc03b370 con 0x7fe70c0fae60 2024-06-15T00:02:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:35.253 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:35.253 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 msgr2=0x7fe6f4036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 0x7fe6f4036ea0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fe700006fd0 tx=0x7fe700006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 msgr2=0x7fe70c10c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc003080 tx=0x7fe6fc009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 shutdown_connections 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6f40349f0 0x7fe6f4036ea0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 --2- 172.21.15.136:0/432423853 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe70c0fae60 0x7fe70c10c4b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 >> 172.21.15.136:0/432423853 conn(0x7fe70c0f65b0 msgr2=0x7fe70c065260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 shutdown_connections 2024-06-15T00:02:35.255 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:35.256+0000 7fe710c78640 1 -- 172.21.15.136:0/432423853 wait complete. 2024-06-15T00:02:36.100 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:35 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/432423853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:37.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:37.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:37.415 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:37.473 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:02:37.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:37.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:37.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:37.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:37.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi063", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 -- 172.21.15.136:0/2550060486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 msgr2=0x7f47280febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 --2- 172.21.15.136:0/2550060486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280febc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f471c004850 tx=0x7f471c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 -- 172.21.15.136:0/2550060486 shutdown_connections 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 --2- 172.21.15.136:0/2550060486 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280febc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 -- 172.21.15.136:0/2550060486 >> 172.21.15.136:0/2550060486 conn(0x7f47280f6610 msgr2=0x7f47280f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 -- 172.21.15.136:0/2550060486 shutdown_connections 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.503+0000 7f472eea7640 1 -- 172.21.15.136:0/2550060486 wait complete. 2024-06-15T00:02:39.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472eea7640 1 Processor -- start 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472eea7640 1 -- start start 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472eea7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472eea7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47280fca30 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472cc43640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472cc43640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:51584/0 (socket says 172.21.15.136:51584) 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.504+0000 7f472cc43640 1 -- 172.21.15.136:0/1352146707 learned_addr learned my addr 172.21.15.136:0/1352146707 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f472cc43640 1 -- 172.21.15.136:0/1352146707 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c005040 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f472cc43640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f471c00af90 tx=0x7f471c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471c007a20 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f471c032b00 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47280faac0 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f471c03dbb0 con 0x7f4728065e40 2024-06-15T00:02:39.527 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.505+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47280faf30 con 0x7f4728065e40 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.506+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f471c030020 con 0x7f4728065e40 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.506+0000 7f47257fa640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 0x7f4718036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.506+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f471c067ff0 con 0x7f4728065e40 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.506+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472805ead0 con 0x7f4728065e40 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.507+0000 7f4727fff640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 0x7f4718036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.507+0000 7f4727fff640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 0x7f4718036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4714006fd0 tx=0x7f4714006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:39.528 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.511+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f471c033420 con 0x7f4728065e40 2024-06-15T00:02:39.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.753+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4728101d10 con 0x7f4728065e40 2024-06-15T00:02:39.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.754+0000 7f47257fa640 1 -- 172.21.15.136:0/1352146707 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f471c045390 con 0x7f4728065e40 2024-06-15T00:02:39.753 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:39.753 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:39.753 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:39.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 msgr2=0x7f4718036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:39.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 0x7f4718036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4714006fd0 tx=0x7f4714006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 msgr2=0x7f47280fc4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f471c00af90 tx=0x7f471c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 shutdown_connections 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f47180349f0 0x7f4718036ea0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 --2- 172.21.15.136:0/1352146707 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4728065e40 0x7f47280fc4f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 >> 172.21.15.136:0/1352146707 conn(0x7f47280f6610 msgr2=0x7f47280f6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 shutdown_connections 2024-06-15T00:02:39.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:39.756+0000 7f472eea7640 1 -- 172.21.15.136:0/1352146707 wait complete. 2024-06-15T00:02:40.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:40.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:40.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:40.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd/host:smithi136", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: Updating smithi063:/etc/ceph/ceph.conf 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: Updating smithi136:/etc/ceph/ceph.conf 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/1352146707' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:40.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:41.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: Updating smithi136:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:41.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: Updating smithi136:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: Updating smithi136:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: Updating smithi063:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: Updating smithi063:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi136", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi136", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-15T00:02:41.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:02:41.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:41.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:41.764 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:42.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:42 smithi063 ceph-mon[28882]: Updating smithi063:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:02:42.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:42 smithi063 ceph-mon[28882]: Deploying daemon crash.smithi136 on smithi136 2024-06-15T00:02:43.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 -- 172.21.15.136:0/3067398363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 msgr2=0x7f24b4101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 --2- 172.21.15.136:0/3067398363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b4101210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f24a4004850 tx=0x7f24a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 -- 172.21.15.136:0/3067398363 shutdown_connections 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 --2- 172.21.15.136:0/3067398363 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b4101210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 -- 172.21.15.136:0/3067398363 >> 172.21.15.136:0/3067398363 conn(0x7f24b4070e10 msgr2=0x7f24b4071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 -- 172.21.15.136:0/3067398363 shutdown_connections 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.432+0000 7f24b95f5640 1 -- 172.21.15.136:0/3067398363 wait complete. 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.433+0000 7f24b95f5640 1 Processor -- start 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.433+0000 7f24b95f5640 1 -- start start 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.433+0000 7f24b95f5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.433+0000 7f24b95f5640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b410da00 con 0x7f24b4100e40 2024-06-15T00:02:43.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b2ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:58888/0 (socket says 172.21.15.136:58888) 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b2ffd640 1 -- 172.21.15.136:0/2331942261 learned_addr learned my addr 172.21.15.136:0/2331942261 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b2ffd640 1 -- 172.21.15.136:0/2331942261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a4005040 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b2ffd640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f24a4003080 tx=0x7f24a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a402cf10 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b410dc00 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24a4033ae0 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24a403db60 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.434+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b410dff0 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.435+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b410e2f0 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.435+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f24a4030020 con 0x7f24b4100e40 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.435+0000 7f24abfff640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 0x7f24a0036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:43.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.436+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f24a4066a30 con 0x7f24b4100e40 2024-06-15T00:02:43.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.436+0000 7f24aa7fc640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 0x7f24a0036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:43.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.436+0000 7f24aa7fc640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 0x7f24a0036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f249c006fd0 tx=0x7f249c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:43.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.441+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f24a4044520 con 0x7f24b4100e40 2024-06-15T00:02:43.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.681+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24b40fc5b0 con 0x7f24b4100e40 2024-06-15T00:02:43.680 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.682+0000 7f24abfff640 1 -- 172.21.15.136:0/2331942261 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f24a4044340 con 0x7f24b4100e40 2024-06-15T00:02:43.681 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:43.681 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:43.681 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:43.682 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 msgr2=0x7f24a0036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 0x7f24a0036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f249c006fd0 tx=0x7f249c006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 msgr2=0x7f24b410d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f24a4003080 tx=0x7f24a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 shutdown_connections 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f24a00349f0 0x7f24a0036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 --2- 172.21.15.136:0/2331942261 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f24b4100e40 0x7f24b410d4c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 >> 172.21.15.136:0/2331942261 conn(0x7f24b4070e10 msgr2=0x7f24b40f9c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 shutdown_connections 2024-06-15T00:02:43.683 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:43.684+0000 7f24b95f5640 1 -- 172.21.15.136:0/2331942261 wait complete. 2024-06-15T00:02:44.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:43 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2331942261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:45.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:45.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:45 smithi063 ceph-mon[28882]: Deploying daemon node-exporter.smithi136 on smithi136 2024-06-15T00:02:45.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:45.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:45.850 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:02:48.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 -- 172.21.15.136:0/799207944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 msgr2=0x7fb47c101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:48.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 --2- 172.21.15.136:0/799207944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c101230 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb478004850 tx=0x7fb478009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 -- 172.21.15.136:0/799207944 shutdown_connections 2024-06-15T00:02:48.498 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 --2- 172.21.15.136:0/799207944 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c101230 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 -- 172.21.15.136:0/799207944 >> 172.21.15.136:0/799207944 conn(0x7fb47c0f6a70 msgr2=0x7fb47c0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 -- 172.21.15.136:0/799207944 shutdown_connections 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.412+0000 7fb4837d8640 1 -- 172.21.15.136:0/799207944 wait complete. 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb4837d8640 1 Processor -- start 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb4837d8640 1 -- start start 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb4837d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb4837d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0f8f40 con 0x7fb47c100e60 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb481574640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb481574640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:58904/0 (socket says 172.21.15.136:58904) 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.413+0000 7fb481574640 1 -- 172.21.15.136:0/486409713 learned_addr learned my addr 172.21.15.136:0/486409713 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb481574640 1 -- 172.21.15.136:0/486409713 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478005040 con 0x7fb47c100e60 2024-06-15T00:02:48.499 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb481574640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb478003080 tx=0x7fb478009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb47802cf10 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c0f74b0 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb478033ae0 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb47803db60 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.414+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c0f7980 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.415+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7fb478030020 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.415+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c0f9080 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.415+0000 7fb471ffb640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 0x7fb468036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.415+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fb478066af0 con 0x7fb47c100e60 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.416+0000 7fb480d73640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 0x7fb468036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.416+0000 7fb480d73640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 0x7fb468036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb46c006fd0 tx=0x7fb46c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:48.500 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.420+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb478044520 con 0x7fb47c100e60 2024-06-15T00:02:48.651 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.653+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb47c0f8600 con 0x7fb47c100e60 2024-06-15T00:02:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.653+0000 7fb471ffb640 1 -- 172.21.15.136:0/486409713 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7fb478044340 con 0x7fb47c100e60 2024-06-15T00:02:48.652 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:48.652 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:48.652 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:48.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.655+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 msgr2=0x7fb468036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:48.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.655+0000 7fb4837d8640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 0x7fb468036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb46c006fd0 tx=0x7fb46c006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.655+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 msgr2=0x7fb47c0f6f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:48.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.655+0000 7fb4837d8640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fb478003080 tx=0x7fb478009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.654 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 shutdown_connections 2024-06-15T00:02:48.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb4680349f0 0x7fb468036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 --2- 172.21.15.136:0/486409713 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb47c100e60 0x7fb47c0f6f10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:48.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 >> 172.21.15.136:0/486409713 conn(0x7fb47c0f6a70 msgr2=0x7fb47c065260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:48.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 shutdown_connections 2024-06-15T00:02:48.655 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:48.656+0000 7fb4837d8640 1 -- 172.21.15.136:0/486409713 wait complete. 2024-06-15T00:02:48.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:48 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/486409713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:50.199 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:50.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:50.445 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi136.frxuxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi136.frxuxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:02:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:50 smithi063 ceph-mon[28882]: Deploying daemon mgr.smithi136.frxuxu on smithi136 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.518+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3321161122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 msgr2=0x7f1cac102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.518+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3321161122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac102310 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c004850 tx=0x7f1c9c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3321161122 shutdown_connections 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3321161122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac102310 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3321161122 >> 172.21.15.136:0/3321161122 conn(0x7f1cac0f7b10 msgr2=0x7f1cac0f9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3321161122 shutdown_connections 2024-06-15T00:02:52.518 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3321161122 wait complete. 2024-06-15T00:02:52.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 Processor -- start 2024-06-15T00:02:52.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.519+0000 7f1cb09b0640 1 -- start start 2024-06-15T00:02:52.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1cb09b0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:52.519 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1cb09b0640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac112a70 con 0x7f1cac101f40 2024-06-15T00:02:52.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1caa59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:52.520 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1caa59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:58926/0 (socket says 172.21.15.136:58926) 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1caa59c640 1 -- 172.21.15.136:0/3399819218 learned_addr learned my addr 172.21.15.136:0/3399819218 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1caa59c640 1 -- 172.21.15.136:0/3399819218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c005040 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.520+0000 7f1caa59c640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c003080 tx=0x7f1c9c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.521+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c9c02cf10 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.521+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c9c033ae0 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.521+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac112c70 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.521+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c9c03db60 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.521+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac113060 con 0x7f1cac101f40 2024-06-15T00:02:52.521 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.522+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f1c9c030020 con 0x7f1cac101f40 2024-06-15T00:02:52.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.522+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac0fd6b0 con 0x7f1cac101f40 2024-06-15T00:02:52.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.522+0000 7f1ca37fe640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 0x7f1c98036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:52.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.522+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f1c9c066bb0 con 0x7f1cac101f40 2024-06-15T00:02:52.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.522+0000 7f1ca1d9b640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 0x7f1c98036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:52.522 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.523+0000 7f1ca1d9b640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 0x7f1c98036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006fd0 tx=0x7f1c94006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:52.526 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.527+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1c9c0323d0 con 0x7f1cac101f40 2024-06-15T00:02:52.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.764+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cac106210 con 0x7f1cac101f40 2024-06-15T00:02:52.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.764+0000 7f1ca37fe640 1 -- 172.21.15.136:0/3399819218 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f1c9c03a2d0 con 0x7f1cac101f40 2024-06-15T00:02:52.763 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:52.763 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:52.764 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 msgr2=0x7f1c98036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 0x7f1c98036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006fd0 tx=0x7f1c94006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 msgr2=0x7f1cac112530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c003080 tx=0x7f1c9c009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 shutdown_connections 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1c980349f0 0x7f1c98036ea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 --2- 172.21.15.136:0/3399819218 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1cac101f40 0x7f1cac112530 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:52.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.766+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 >> 172.21.15.136:0/3399819218 conn(0x7f1cac0f7b10 msgr2=0x7f1cac0f9300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:52.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.767+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 shutdown_connections 2024-06-15T00:02:52.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:52.767+0000 7f1cb09b0640 1 -- 172.21.15.136:0/3399819218 wait complete. 2024-06-15T00:02:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:52 smithi063 ceph-mon[28882]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:52 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/3399819218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:02:54.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:54.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:54.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:54.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:54.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:02:54.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:02:54.431 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:54 smithi063 ceph-mon[28882]: Deploying daemon mon.smithi136 on smithi136 2024-06-15T00:02:54.630 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:02:54.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:02:55.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:55 smithi063 ceph-mon[28882]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:55.698 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:02:57.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:57 smithi063 ceph-mon[28882]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:02:57.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:57 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:02:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 -- 172.21.15.136:0/1313543153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 msgr2=0x7f06b4102390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1313543153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b4102390 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f06a4004850 tx=0x7f06a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:57.802 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 -- 172.21.15.136:0/1313543153 shutdown_connections 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1313543153 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b4102390 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 -- 172.21.15.136:0/1313543153 >> 172.21.15.136:0/1313543153 conn(0x7f06b40714f0 msgr2=0x7f06b40718f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 -- 172.21.15.136:0/1313543153 shutdown_connections 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.775+0000 7f06b9e56640 1 -- 172.21.15.136:0/1313543153 wait complete. 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b9e56640 1 Processor -- start 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b9e56640 1 -- start start 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b9e56640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b9e56640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b4112b00 con 0x7f06b4101fc0 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.136:42732/0 (socket says 172.21.15.136:42732) 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.776+0000 7f06b37fe640 1 -- 172.21.15.136:0/1254215285 learned_addr learned my addr 172.21.15.136:0/1254215285 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06b37fe640 1 -- 172.21.15.136:0/1254215285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a4005040 con 0x7f06b4101fc0 2024-06-15T00:02:57.803 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06b37fe640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f06a40048b0 tx=0x7f06a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06a4033700 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b4112d00 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f06a4033d10 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06a403dd00 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.777+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b41130f0 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.778+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 17) v1 ==== 42059+0+0 (secure 0 0 0) 0x7f06a4030020 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.778+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b40fd8d0 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.778+0000 7f06abfff640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 0x7f06a0036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.778+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f06a4066ff0 con 0x7f06b4101fc0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.779+0000 7f06b2ffd640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 0x7f06a0036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.779+0000 7f06b2ffd640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 0x7f06a0036ea0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f069c006fd0 tx=0x7f069c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:02:57.804 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:57.783+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f06a4045d60 con 0x7f06b4101fc0 2024-06-15T00:02:58.037 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.039+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06b4106260 con 0x7f06b4101fc0 2024-06-15T00:02:58.038 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.039+0000 7f06abfff640 1 -- 172.21.15.136:0/1254215285 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+739 (secure 0 0 0) 0x7f06a4045b80 con 0x7f06b4101fc0 2024-06-15T00:02:58.038 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:02:58.038 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":1,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:00:27.387292Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:02:58.038 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 1 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.041+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 msgr2=0x7f06a0036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.041+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 0x7f06a0036ea0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f069c006fd0 tx=0x7f069c006db0 comp rx=0 tx=0).stop 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.041+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 msgr2=0x7f06b41125c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.041+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f06a40048b0 tx=0x7f06a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.041+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 shutdown_connections 2024-06-15T00:02:58.040 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.042+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f06a00349f0 0x7f06a0036ea0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:58.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.042+0000 7f06b9e56640 1 --2- 172.21.15.136:0/1254215285 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f06b4101fc0 0x7f06b41125c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:02:58.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.042+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 >> 172.21.15.136:0/1254215285 conn(0x7f06b40714f0 msgr2=0x7f06b4102d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:02:58.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.042+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 shutdown_connections 2024-06-15T00:02:58.041 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:02:58.042+0000 7f06b9e56640 1 -- 172.21.15.136:0/1254215285 wait complete. 2024-06-15T00:02:58.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: Failed to load ceph-mgr modules: k8sevents 2024-06-15T00:02:58.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: Standby manager daemon smithi136.frxuxu started 2024-06-15T00:02:58.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: from='mgr.? 172.21.15.136:0/1934980152' entity='mgr.smithi136.frxuxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi136.frxuxu/crt"}]: dispatch 2024-06-15T00:02:58.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: from='mgr.? 172.21.15.136:0/1934980152' entity='mgr.smithi136.frxuxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-15T00:02:58.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: from='mgr.? 172.21.15.136:0/1934980152' entity='mgr.smithi136.frxuxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi136.frxuxu/key"}]: dispatch 2024-06-15T00:02:58.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: from='mgr.? 172.21.15.136:0/1934980152' entity='mgr.smithi136.frxuxu' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-15T00:02:58.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:02:58 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/1254215285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:03:00.042 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:03:00.042 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:03:00.302 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:03:03.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:03:03.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mon.smithi063 calling monitor election 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mon.smithi136 calling monitor election 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:03.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mon.smithi063 is new leader, mons smithi063,smithi136 in quorum (ranks 0,1) 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: monmap e2: 2 mons at {smithi063=[v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0],smithi136=[v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: fsmap 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mgrmap e18: smithi063.dhizav(active, since 31s), standbys: smithi136.frxuxu 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mgr module k8sevents crashed in daemon mgr.smithi063.dhizav on host smithi063 at 2024-06-15T00:00:41.419873Z 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mgr module k8sevents crashed in daemon mgr.smithi063.dhizav on host smithi063 at 2024-06-15T00:00:54.940824Z 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: mgr module k8sevents crashed in daemon mgr.smithi063.dhizav on host smithi063 at 2024-06-15T00:01:24.816520Z 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:03.931 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:04.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:03 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 -- 172.21.15.136:0/736159638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90102050 msgr2=0x7fba7c005540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 --2- 172.21.15.136:0/736159638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90102050 0x7fba7c005540 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fba8c007d80 tx=0x7fba8c00b040 comp rx=0 tx=0).stop 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 -- 172.21.15.136:0/736159638 shutdown_connections 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 --2- 172.21.15.136:0/736159638 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90102050 0x7fba7c005540 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 -- 172.21.15.136:0/736159638 >> 172.21.15.136:0/736159638 conn(0x7fba90070f20 msgr2=0x7fba90071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 -- 172.21.15.136:0/736159638 shutdown_connections 2024-06-15T00:03:04.873 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.874+0000 7fba9779d640 1 -- 172.21.15.136:0/736159638 wait complete. 2024-06-15T00:03:04.874 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:04 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:04.874 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:04 smithi136 ceph-mon[35883]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:04.874 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:04 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:04.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 Processor -- start 2024-06-15T00:03:04.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 -- start start 2024-06-15T00:03:04.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:04.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 0x7fba9010c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:04.875 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba901130e0 con 0x7fba90112b60 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba9779d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90113220 con 0x7fba90102050 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:53000/0 (socket says 172.21.15.136:53000) 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 -- 172.21.15.136:0/2516547780 learned_addr learned my addr 172.21.15.136:0/2516547780 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 -- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 msgr2=0x7fba90112620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 -- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 msgr2=0x7fba90112620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba95539640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.875+0000 7fba94d38640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 0x7fba9010c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba94d38640 1 -- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 msgr2=0x7fba90112620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba94d38640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba94d38640 1 -- 172.21.15.136:0/2516547780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c005040 con 0x7fba90112b60 2024-06-15T00:03:04.876 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba94d38640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 0x7fba9010c580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b580 tx=0x7fba800090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:04.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba80011690 con 0x7fba90112b60 2024-06-15T00:03:04.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fba80011ca0 con 0x7fba90112b60 2024-06-15T00:03:04.877 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9010cb80 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fba80010e30 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.876+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9010d120 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.878+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba9010d400 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.879+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fba800105e0 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.879+0000 7fba85ffb640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 0x7fba74067960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.879+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fba800819b0 con 0x7fba90112b60 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.879+0000 7fba95539640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 0x7fba74067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:04.878 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.880+0000 7fba95539640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 0x7fba74067960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba8c004010 tx=0x7fba8c03c040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:04.882 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:04.884+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fba80050c80 con 0x7fba90112b60 2024-06-15T00:03:04.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:04 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:04.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:04 smithi063 ceph-mon[28882]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:04.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:04 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:03:05.113 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.115+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba90103cd0 con 0x7fba90112b60 2024-06-15T00:03:05.114 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.116+0000 7fba85ffb640 1 -- 172.21.15.136:0/2516547780 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1017 (secure 0 0 0) 0x7fba80015090 con 0x7fba90112b60 2024-06-15T00:03:05.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-06-15T00:03:05.114 INFO:teuthology.orchestra.run.smithi136.stdout:{"epoch":2,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:02:58.544437Z","created":"2024-06-15T00:00:27.387292Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi063","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:3300","nonce":0},{"type":"v1","addr":"172.21.15.63:6789","nonce":0}]},"addr":"172.21.15.63:6789/0","public_addr":"172.21.15.63:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi136","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:3300","nonce":0},{"type":"v1","addr":"172.21.15.136:6789","nonce":0}]},"addr":"172.21.15.136:6789/0","public_addr":"172.21.15.136:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-15T00:03:05.114 INFO:teuthology.orchestra.run.smithi136.stderr:dumped monmap epoch 2 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 msgr2=0x7fba74067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 0x7fba74067960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba8c004010 tx=0x7fba8c03c040 comp rx=0 tx=0).stop 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 msgr2=0x7fba9010c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 0x7fba9010c580 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fba8000b580 tx=0x7fba800090f0 comp rx=0 tx=0).stop 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 shutdown_connections 2024-06-15T00:03:05.116 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fba740654b0 0x7fba74067960 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:05.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fba90112b60 0x7fba9010c580 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:05.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 --2- 172.21.15.136:0/2516547780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba90102050 0x7fba90112620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:05.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 >> 172.21.15.136:0/2516547780 conn(0x7fba90070f20 msgr2=0x7fba900fa3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:05.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 shutdown_connections 2024-06-15T00:03:05.117 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:03:05.118+0000 7fba9779d640 1 -- 172.21.15.136:0/2516547780 wait complete. 2024-06-15T00:03:05.733 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:05 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/2516547780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:03:05.758 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-15T00:03:05.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-15T00:03:05.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:05 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2516547780' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:03:06.034 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:07.413 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: Updating smithi063:/etc/ceph/ceph.conf 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: Updating smithi136:/etc/ceph/ceph.conf 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:07.414 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: Updating smithi063:/etc/ceph/ceph.conf 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: Updating smithi136:/etc/ceph/ceph.conf 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:03:07.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:03:07.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:07.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 -- 172.21.15.63:0/3424884070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc101f60 msgr2=0x7f67cc102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:07.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 --2- 172.21.15.63:0/3424884070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc101f60 0x7f67cc102330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f67c8004850 tx=0x7f67c8009da0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 -- 172.21.15.63:0/3424884070 shutdown_connections 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 --2- 172.21.15.63:0/3424884070 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc101f60 0x7f67cc102330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 -- 172.21.15.63:0/3424884070 >> 172.21.15.63:0/3424884070 conn(0x7f67cc0f7b30 msgr2=0x7f67cc0f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 -- 172.21.15.63:0/3424884070 shutdown_connections 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.722+0000 7f67d379f640 1 -- 172.21.15.63:0/3424884070 wait complete. 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 Processor -- start 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 -- start start 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc101f60 0x7f67cc111500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc112000 con 0x7f67cc111a40 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d379f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc112170 con 0x7f67cc101f60 2024-06-15T00:03:07.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39338/0 (socket says 172.21.15.63:39338) 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 -- 172.21.15.63:0/1988936581 learned_addr learned my addr 172.21.15.63:0/1988936581 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 -- 172.21.15.63:0/1988936581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc101f60 msgr2=0x7f67cc111500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc101f60 0x7f67cc111500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.723+0000 7f67d0d3a640 1 -- 172.21.15.63:0/1988936581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c8005040 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67d0d3a640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00b790 tx=0x7f67bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67bc00c730 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67bc00cd40 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc10cc30 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67bc0133a0 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.724+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc10d150 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.725+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc10d480 con 0x7f67cc111a40 2024-06-15T00:03:07.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.725+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f67bc00c890 con 0x7f67cc111a40 2024-06-15T00:03:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.726+0000 7f67c1ffb640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 0x7f67b80679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.726+0000 7f67d153b640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 0x7f67b80679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.726+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f67bc080db0 con 0x7f67cc111a40 2024-06-15T00:03:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.726+0000 7f67d153b640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 0x7f67b80679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f67c802ab60 tx=0x7f67c8009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:07.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.730+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f67bc04cdf0 con 0x7f67cc111a40 2024-06-15T00:03:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.912+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f67cc1061a0 con 0x7f67cc111a40 2024-06-15T00:03:07.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.913+0000 7f67c1ffb640 1 -- 172.21.15.63:0/1988936581 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+225 (secure 0 0 0) 0x7f67bc019090 con 0x7f67cc111a40 2024-06-15T00:03:07.911 INFO:teuthology.orchestra.run.smithi063.stdout:# minimal ceph.conf for 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:03:07.911 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2024-06-15T00:03:07.911 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 25f32e20-2aaa-11ef-bc9d-c7b262605968 2024-06-15T00:03:07.912 INFO:teuthology.orchestra.run.smithi063.stdout: mon_host = [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] 2024-06-15T00:03:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 msgr2=0x7f67b80679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 0x7f67b80679a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f67c802ab60 tx=0x7f67c8009da0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 msgr2=0x7f67cc10c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00b790 tx=0x7f67bc0091d0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 shutdown_connections 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f67b80654f0 0x7f67b80679a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f67cc111a40 0x7f67cc10c630 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 --2- 172.21.15.63:0/1988936581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67cc101f60 0x7f67cc111500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 >> 172.21.15.63:0/1988936581 conn(0x7f67cc0f7b30 msgr2=0x7f67cc0f96e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 shutdown_connections 2024-06-15T00:03:07.914 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:07.915+0000 7f67d379f640 1 -- 172.21.15.63:0/1988936581 wait complete. 2024-06-15T00:03:08.322 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:08 smithi063 ceph-mon[28882]: Updating smithi136:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:03:08.322 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:08 smithi063 ceph-mon[28882]: Updating smithi063:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:03:08.322 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:08 smithi063 ceph-mon[28882]: Reconfiguring mon.smithi063 (unknown last config time)... 2024-06-15T00:03:08.322 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:08 smithi063 ceph-mon[28882]: Reconfiguring daemon mon.smithi063 on smithi063 2024-06-15T00:03:08.322 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:08 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1988936581' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:08.486 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:08 smithi136 ceph-mon[35883]: Updating smithi136:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:03:08.486 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:08 smithi136 ceph-mon[35883]: Updating smithi063:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:03:08.487 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:08 smithi136 ceph-mon[35883]: Reconfiguring mon.smithi063 (unknown last config time)... 2024-06-15T00:03:08.487 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:08 smithi136 ceph-mon[35883]: Reconfiguring daemon mon.smithi063 on smithi063 2024-06-15T00:03:08.487 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:08 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1988936581' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.dhizav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:03:09.398 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:09.421 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-15T00:03:09.421 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:03:09.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.conf 2024-06-15T00:03:09.448 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:03:09.448 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:03:09.514 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:03:09.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.conf 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi063.dhizav", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:03:09.537 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:09.541 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:03:09.542 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:03:09.607 INFO:tasks.cephadm:Deploying OSDs... 2024-06-15T00:03:09.607 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:03:09.607 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T00:03:09.624 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T00:03:09.624 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-06-15 00:02:25.444293426 +0000 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-06-14 23:56:41.540868868 +0000 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-06-14 23:56:41.540868868 +0000 2024-06-15T00:03:09.681 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-06-14 23:56:41.540868868 +0000 2024-06-15T00:03:09.681 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T00:03:09.746 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-06-15T00:03:09.746 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-06-15T00:03:09.746 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000199438 s, 2.6 MB/s 2024-06-15T00:03:09.747 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T00:03:09.804 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 2024-06-15T00:03:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T00:03:09.860 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:09.860 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-15T00:03:09.860 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:09.860 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:09.861 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-06-15 00:02:25.445293418 +0000 2024-06-15T00:03:09.861 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-06-14 23:56:41.897866194 +0000 2024-06-15T00:03:09.861 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-06-14 23:56:41.897866194 +0000 2024-06-15T00:03:09.861 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-06-14 23:56:41.897866194 +0000 2024-06-15T00:03:09.861 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T00:03:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-06-15T00:03:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-06-15T00:03:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000174233 s, 2.9 MB/s 2024-06-15T00:03:09.925 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T00:03:09.982 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 2024-06-15T00:03:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T00:03:10.038 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.038 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-15T00:03:10.038 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.038 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.039 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-06-15 00:02:25.445293418 +0000 2024-06-15T00:03:10.039 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-06-14 23:56:42.222863760 +0000 2024-06-15T00:03:10.039 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-06-14 23:56:42.222863760 +0000 2024-06-15T00:03:10.039 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-06-14 23:56:42.222863760 +0000 2024-06-15T00:03:10.039 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T00:03:10.102 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-06-15T00:03:10.102 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-06-15T00:03:10.102 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000200091 s, 2.6 MB/s 2024-06-15T00:03:10.103 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T00:03:10.160 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2024-06-15 00:02:25.446293410 +0000 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2024-06-14 23:56:42.551861296 +0000 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2024-06-14 23:56:42.551861296 +0000 2024-06-15T00:03:10.216 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: 2024-06-14 23:56:42.551861296 +0000 2024-06-15T00:03:10.217 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T00:03:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2024-06-15T00:03:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2024-06-15T00:03:10.280 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000187654 s, 2.7 MB/s 2024-06-15T00:03:10.281 DEBUG:teuthology.orchestra.run.smithi063:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T00:03:10.337 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:03:10.337 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T00:03:10.355 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T00:03:10.355 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-06-15T00:03:10.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:10 smithi063 ceph-mon[28882]: Reconfiguring mgr.smithi063.dhizav (unknown last config time)... 2024-06-15T00:03:10.403 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:10 smithi063 ceph-mon[28882]: Reconfiguring daemon mgr.smithi063.dhizav on smithi063 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-06-15 00:02:57.521724442 +0000 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-06-14 23:56:35.609443447 +0000 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-06-14 23:56:35.609443447 +0000 2024-06-15T00:03:10.412 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-06-14 23:56:35.609443447 +0000 2024-06-15T00:03:10.413 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T00:03:10.475 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:10 smithi136 ceph-mon[35883]: Reconfiguring mgr.smithi063.dhizav (unknown last config time)... 2024-06-15T00:03:10.476 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:10 smithi136 ceph-mon[35883]: Reconfiguring daemon mgr.smithi063.dhizav on smithi063 2024-06-15T00:03:10.477 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-06-15T00:03:10.477 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-06-15T00:03:10.478 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000203436 s, 2.5 MB/s 2024-06-15T00:03:10.479 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T00:03:10.536 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 689 Links: 1 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-06-15 00:02:57.522724421 +0000 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-06-14 23:56:35.940436256 +0000 2024-06-15T00:03:10.592 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-06-14 23:56:35.940436256 +0000 2024-06-15T00:03:10.593 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-06-14 23:56:35.940436256 +0000 2024-06-15T00:03:10.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T00:03:10.655 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-06-15T00:03:10.656 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-06-15T00:03:10.656 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.00026963 s, 1.9 MB/s 2024-06-15T00:03:10.657 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T00:03:10.713 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.770 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-06-15 00:02:57.522724421 +0000 2024-06-15T00:03:10.771 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-06-14 23:56:36.268429130 +0000 2024-06-15T00:03:10.771 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-06-14 23:56:36.268429130 +0000 2024-06-15T00:03:10.771 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-06-14 23:56:36.268429130 +0000 2024-06-15T00:03:10.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T00:03:10.833 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-06-15T00:03:10.833 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-06-15T00:03:10.834 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000204225 s, 2.5 MB/s 2024-06-15T00:03:10.835 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T00:03:10.892 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-06-15 00:02:57.522724421 +0000 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-06-14 23:56:36.608421744 +0000 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-06-14 23:56:36.608421744 +0000 2024-06-15T00:03:10.948 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-06-14 23:56:36.608421744 +0000 2024-06-15T00:03:10.949 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T00:03:11.011 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-06-15T00:03:11.011 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-06-15T00:03:11.011 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000228692 s, 2.2 MB/s 2024-06-15T00:03:11.013 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T00:03:11.069 INFO:tasks.cephadm:Deploying osd.0 on smithi063 with /dev/vg_nvme/lv_4... 2024-06-15T00:03:11.069 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-15T00:03:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:11 smithi063 ceph-mon[28882]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:03:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:11.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:11 smithi136 ceph-mon[35883]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:11.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:11.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:11.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi063", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:03:11.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:11.864 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:03:12.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:12 smithi063 ceph-mon[28882]: Reconfiguring crash.smithi063 (monmap changed)... 2024-06-15T00:03:12.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:12 smithi063 ceph-mon[28882]: Reconfiguring daemon crash.smithi063 on smithi063 2024-06-15T00:03:12.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:12 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:12.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:12 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:12.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:12 smithi136 ceph-mon[35883]: Reconfiguring crash.smithi063 (monmap changed)... 2024-06-15T00:03:12.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:12 smithi136 ceph-mon[35883]: Reconfiguring daemon crash.smithi063 on smithi063 2024-06-15T00:03:12.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:12 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:12.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:12 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:13.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:13 smithi063 ceph-mon[28882]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-06-15T00:03:13.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:13 smithi063 ceph-mon[28882]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-06-15T00:03:13.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:13 smithi063 ceph-mon[28882]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:13.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:13 smithi136 ceph-mon[35883]: Reconfiguring alertmanager.smithi063 (dependencies changed)... 2024-06-15T00:03:13.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:13 smithi136 ceph-mon[35883]: Reconfiguring daemon alertmanager.smithi063 on smithi063 2024-06-15T00:03:13.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:13 smithi136 ceph-mon[35883]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:14.291 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-15T00:03:14.383 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-15T00:03:14.396 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-06-15T00:03:14.397 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-06-15T00:03:14.397 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116555 s, 900 MB/s 2024-06-15T00:03:14.397 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-06-15T00:03:15.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_4 2024-06-15T00:03:15.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:15 smithi063 ceph-mon[28882]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:15.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:15 smithi136 ceph-mon[35883]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:16.004 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:03:17.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:17 smithi063 ceph-mon[28882]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:17.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:17.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:17 smithi136 ceph-mon[35883]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:17.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:17.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 -- 172.21.15.63:0/3506727091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fb140 msgr2=0x7f0b240fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:17.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/3506727091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fb140 0x7f0b240fb550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b200048d0 tx=0x7f0b20009da0 comp rx=0 tx=0).stop 2024-06-15T00:03:17.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 -- 172.21.15.63:0/3506727091 shutdown_connections 2024-06-15T00:03:17.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/3506727091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fc410 0x7f0b240fc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/3506727091 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fb140 0x7f0b240fb550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 -- 172.21.15.63:0/3506727091 >> 172.21.15.63:0/3506727091 conn(0x7f0b240f68e0 msgr2=0x7f0b240f8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 -- 172.21.15.63:0/3506727091 shutdown_connections 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.854+0000 7f0b2b34d640 1 -- 172.21.15.63:0/3506727091 wait complete. 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 Processor -- start 2024-06-15T00:03:17.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 -- start start 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 0x7f0b2410bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2410cfe0 con 0x7f0b240fc410 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b2b34d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2410d150 con 0x7f0b240fb140 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b288e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b288e8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50286/0 (socket says 172.21.15.63:50286) 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b288e8640 1 -- 172.21.15.63:0/2385911828 learned_addr learned my addr 172.21.15.63:0/2385911828 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:03:17.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.855+0000 7f0b290e9640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 0x7f0b2410bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b288e8640 1 -- 172.21.15.63:0/2385911828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 msgr2=0x7f0b2410bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b288e8640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 0x7f0b2410bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b288e8640 1 -- 172.21.15.63:0/2385911828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20005040 con 0x7f0b240fc410 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b290e9640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 0x7f0b2410bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b288e8640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400b790 tx=0x7f0b140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b1400c700 con 0x7f0b240fc410 2024-06-15T00:03:17.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b1400cd10 con 0x7f0b240fc410 2024-06-15T00:03:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2410d430 con 0x7f0b240fc410 2024-06-15T00:03:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b14013370 con 0x7f0b240fc410 2024-06-15T00:03:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.856+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2410d950 con 0x7f0b240fc410 2024-06-15T00:03:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.857+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b2405ead0 con 0x7f0b240fc410 2024-06-15T00:03:17.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.857+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7f0b14012070 con 0x7f0b240fc410 2024-06-15T00:03:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.858+0000 7f0b19ffb640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 0x7f0b10069a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.858+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0b14081310 con 0x7f0b240fc410 2024-06-15T00:03:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.858+0000 7f0b290e9640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 0x7f0b10069a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.859+0000 7f0b290e9640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 0x7f0b10069a60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004540 tx=0x7f0b20009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:17.861 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:17.862+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0b140505b0 con 0x7f0b240fc410 2024-06-15T00:03:18.047 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:18.049+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b2410dc20 con 0x7f0b100675b0 2024-06-15T00:03:18.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:18 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:03:18.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:18 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:03:18.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:18 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:18.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:18 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:03:18.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:18 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:03:18.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:18 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:19.295 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:19 smithi063 ceph-mon[28882]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:19.296 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:19 smithi063 ceph-mon[28882]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:03:19.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:19 smithi136 ceph-mon[35883]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:19.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:19 smithi136 ceph-mon[35883]: from='client.14248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:03:21.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:21 smithi063 ceph-mon[28882]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:21.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:21 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:21.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:21 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:21.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:21 smithi136 ceph-mon[35883]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:21.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:21 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:21.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:21 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:22.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-06-15T00:03:22.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-06-15T00:03:22.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2151855175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d"}]: dispatch 2024-06-15T00:03:22.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2151855175' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d"}]': finished 2024-06-15T00:03:22.280 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: osdmap e6: 1 total, 0 up, 1 in 2024-06-15T00:03:22.281 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:22 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:22.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: Reconfiguring grafana.smithi063 (dependencies changed)... 2024-06-15T00:03:22.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: Reconfiguring daemon grafana.smithi063 on smithi063 2024-06-15T00:03:22.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2151855175' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d"}]: dispatch 2024-06-15T00:03:22.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2151855175' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d"}]': finished 2024-06-15T00:03:22.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: osdmap e6: 1 total, 0 up, 1 in 2024-06-15T00:03:22.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:22 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:23.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:23 smithi063 ceph-mon[28882]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:23.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:23 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1859672033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:03:23.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:23 smithi136 ceph-mon[35883]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:23.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:23 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1859672033' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:03:25.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:25 smithi063 ceph-mon[28882]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:25.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:25 smithi136 ceph-mon[35883]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:27.217 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:27 smithi063 ceph-mon[28882]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:27.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:27 smithi136 ceph-mon[35883]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:29.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:29 smithi063 ceph-mon[28882]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:29.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:29 smithi136 ceph-mon[35883]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:31.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:31 smithi063 ceph-mon[28882]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:31.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:31.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:31.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:31 smithi136 ceph-mon[35883]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:31.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:31.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:32.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:32 smithi063 ceph-mon[28882]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-06-15T00:03:32.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:32 smithi063 ceph-mon[28882]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-06-15T00:03:32.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:32.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-15T00:03:32.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:32.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:32 smithi136 ceph-mon[35883]: Reconfiguring prometheus.smithi063 (dependencies changed)... 2024-06-15T00:03:32.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:32 smithi136 ceph-mon[35883]: Reconfiguring daemon prometheus.smithi063 on smithi063 2024-06-15T00:03:32.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:32.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-15T00:03:32.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:33.354 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:33 smithi063 ceph-mon[28882]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:33.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:33 smithi063 ceph-mon[28882]: Deploying daemon osd.0 on smithi063 2024-06-15T00:03:33.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:33 smithi136 ceph-mon[35883]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:33.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:33 smithi136 ceph-mon[35883]: Deploying daemon osd.0 on smithi063 2024-06-15T00:03:34.494 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:34 smithi063 ceph-mon[28882]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:34.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:34 smithi136 ceph-mon[35883]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:37.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:36 smithi063 ceph-mon[28882]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:37.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:37.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:37.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi136", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:03:37.212 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:37.279 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:36 smithi136 ceph-mon[35883]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:37.280 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:37.280 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:37.280 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi136", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:03:37.280 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: Reconfiguring crash.smithi136 (monmap changed)... 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: Reconfiguring daemon crash.smithi136 on smithi136 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi136.frxuxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:03:38.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:03:38.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:37 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: Reconfiguring crash.smithi136 (monmap changed)... 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: Reconfiguring daemon crash.smithi136 on smithi136 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi136.frxuxu", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:03:38.189 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:37 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:39.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:38 smithi063 ceph-mon[28882]: Reconfiguring mgr.smithi136.frxuxu (monmap changed)... 2024-06-15T00:03:39.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:38 smithi063 ceph-mon[28882]: Reconfiguring daemon mgr.smithi136.frxuxu on smithi136 2024-06-15T00:03:39.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:38 smithi063 ceph-mon[28882]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:38 smithi136 ceph-mon[35883]: Reconfiguring mgr.smithi136.frxuxu (monmap changed)... 2024-06-15T00:03:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:38 smithi136 ceph-mon[35883]: Reconfiguring daemon mgr.smithi136.frxuxu on smithi136 2024-06-15T00:03:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:38 smithi136 ceph-mon[35883]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:40.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: Reconfiguring mon.smithi136 (monmap changed)... 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:40.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:39 smithi063 ceph-mon[28882]: Reconfiguring daemon mon.smithi136 on smithi136 2024-06-15T00:03:40.239 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: Reconfiguring mon.smithi136 (monmap changed)... 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:40.240 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:39 smithi136 ceph-mon[35883]: Reconfiguring daemon mon.smithi136 on smithi136 2024-06-15T00:03:41.264 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:41.264 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.264 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.264 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.265 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:40 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:03:41.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:41.336 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:40 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:42.274 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.275 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:42 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:03:42.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi063.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi063.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi063.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:42.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:42 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:43 smithi063 ceph-mon[28882]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:43.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:43 smithi136 ceph-mon[35883]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:44.940 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.942+0000 7f0b19ffb640 1 -- 172.21.15.63:0/2385911828 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0b2410dc20 con 0x7f0b100675b0 2024-06-15T00:03:44.941 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 0 on host 'smithi063' 2024-06-15T00:03:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 msgr2=0x7f0b10069a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 0x7f0b10069a60 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0b20004540 tx=0x7f0b20009da0 comp rx=0 tx=0).stop 2024-06-15T00:03:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 msgr2=0x7f0b2410c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:44.942 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0b1400b790 tx=0x7f0b140091d0 comp rx=0 tx=0).stop 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 shutdown_connections 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b100675b0 0x7f0b10069a60 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b240fc410 0x7f0b2410c300 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 --2- 172.21.15.63:0/2385911828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b240fb140 0x7f0b2410bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 >> 172.21.15.63:0/2385911828 conn(0x7f0b240f68e0 msgr2=0x7f0b240f80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 shutdown_connections 2024-06-15T00:03:44.943 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:44.944+0000 7f0b2b34d640 1 -- 172.21.15.63:0/2385911828 wait complete. 2024-06-15T00:03:45.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:45 smithi063 ceph-mon[28882]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:45.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:45.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:45.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:45 smithi063 ceph-mon[28882]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-15T00:03:45.510 DEBUG:teuthology.orchestra.run.smithi063:osd.0> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.0.service 2024-06-15T00:03:45.513 INFO:tasks.cephadm:Deploying osd.1 on smithi063 with /dev/vg_nvme/lv_3... 2024-06-15T00:03:45.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-15T00:03:45.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:45 smithi136 ceph-mon[35883]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:45.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:45.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:45.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:45 smithi136 ceph-mon[35883]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-15T00:03:45.838 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: osdmap e7: 1 total, 0 up, 1 in 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:03:47.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:47.180 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0[46684]: 2024-06-15T00:03:46.949+0000 7fcdd6263640 -1 osd.0 0 waiting for initial osdmap 2024-06-15T00:03:47.180 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:03:46 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0[46684]: 2024-06-15T00:03:46.952+0000 7fcdd109a640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:03:47.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:47.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-15T00:03:47.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: osdmap e7: 1 total, 0 up, 1 in 2024-06-15T00:03:47.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:47.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:03:47.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:03:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:47 smithi063 ceph-mon[28882]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:03:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:47 smithi063 ceph-mon[28882]: osdmap e8: 1 total, 0 up, 1 in 2024-06-15T00:03:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:48.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:48.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:47 smithi136 ceph-mon[35883]: from='osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:03:48.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:47 smithi136 ceph-mon[35883]: osdmap e8: 1 total, 0 up, 1 in 2024-06-15T00:03:48.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:48.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:48.986 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-15T00:03:49.070 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-15T00:03:49.083 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-06-15T00:03:49.083 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-06-15T00:03:49.084 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112381 s, 933 MB/s 2024-06-15T00:03:49.084 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-06-15T00:03:49.202 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] boot 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: osdmap e9: 1 total, 1 up, 1 in 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:49.203 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:48 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:49.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: osd.0 [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] boot 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: osdmap e9: 1 total, 1 up, 1 in 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:49.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:48 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:49.748 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_3 2024-06-15T00:03:50.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:49 smithi063 ceph-mon[28882]: osdmap e10: 1 total, 1 up, 1 in 2024-06-15T00:03:50.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:49 smithi136 ceph-mon[35883]: osdmap e10: 1 total, 1 up, 1 in 2024-06-15T00:03:50.977 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:03:51.102 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:50 smithi063 ceph-mon[28882]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:51.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:50 smithi136 ceph-mon[35883]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:53.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 -- 172.21.15.63:0/366417988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fc120 msgr2=0x7ff1940fc570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:53.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 --2- 172.21.15.63:0/366417988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fc120 0x7ff1940fc570 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff17c0048d0 tx=0x7ff17c009da0 comp rx=0 tx=0).stop 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 -- 172.21.15.63:0/366417988 shutdown_connections 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 --2- 172.21.15.63:0/366417988 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fc120 0x7ff1940fc570 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 --2- 172.21.15.63:0/366417988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fb7d0 0x7ff1940fbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 -- 172.21.15.63:0/366417988 >> 172.21.15.63:0/366417988 conn(0x7ff1940f6100 msgr2=0x7ff1940f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 -- 172.21.15.63:0/366417988 shutdown_connections 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.095+0000 7ff199094640 1 -- 172.21.15.63:0/366417988 wait complete. 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 Processor -- start 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 -- start start 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fc120 0x7ff1941120b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940fb190 con 0x7ff1940fb7d0 2024-06-15T00:03:53.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff199094640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940fb300 con 0x7ff1940fc120 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:34104/0 (socket says 172.21.15.63:34104) 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 -- 172.21.15.63:0/745528781 learned_addr learned my addr 172.21.15.63:0/745528781 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff18bfff640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fc120 0x7ff1941120b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 -- 172.21.15.63:0/745528781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fc120 msgr2=0x7ff1941120b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fc120 0x7ff1941120b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.096+0000 7ff192d9d640 1 -- 172.21.15.63:0/745528781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17c005040 con 0x7ff1940fb7d0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff192d9d640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c790 tx=0x7ff1840043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff18400a7a0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff194112650 con 0x7ff1940fb7d0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff18400adb0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1840113a0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.097 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.097+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff194112bf0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.098+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1940f9da0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.098+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7ff18400a900 con 0x7ff1940fb7d0 2024-06-15T00:03:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.099+0000 7ff18b7fe640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 0x7ff180067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:03:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.099+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7ff1840819e0 con 0x7ff1940fb7d0 2024-06-15T00:03:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.099+0000 7ff18bfff640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 0x7ff180067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:03:53.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.099+0000 7ff18bfff640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 0x7ff180067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff17c000c00 tx=0x7ff17c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:03:53.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.103+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff184050a50 con 0x7ff1940fb7d0 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:53.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.180 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:52 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:53.205 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:53.205 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.205 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.205 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:03:53.205 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.206 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:53.206 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:53.206 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:53.206 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:52 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:03:53.292 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:03:53.293+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1941007f0 con 0x7ff1800653a0 2024-06-15T00:03:54.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:53 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi063 2024-06-15T00:03:54.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:53 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi063 to 12864M 2024-06-15T00:03:54.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:53 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:03:54.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:53 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:03:54.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:53 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:54.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:53 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi063 2024-06-15T00:03:54.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:53 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi063 to 12864M 2024-06-15T00:03:54.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:53 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:03:54.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:53 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:03:54.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:53 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:55.147 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:54 smithi063 ceph-mon[28882]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:03:55.147 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:54 smithi063 ceph-mon[28882]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:55.147 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:55.148 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:54 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:54 smithi136 ceph-mon[35883]: from='client.14266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:03:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:54 smithi136 ceph-mon[35883]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:55.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:54 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:55.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:54 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:57.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:56 smithi063 ceph-mon[28882]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:57.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:56 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2797491966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033f1a9-211c-4c35-a2b9-33361d67ddd2"}]: dispatch 2024-06-15T00:03:57.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:56 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2797491966' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a033f1a9-211c-4c35-a2b9-33361d67ddd2"}]': finished 2024-06-15T00:03:57.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:56 smithi063 ceph-mon[28882]: osdmap e11: 2 total, 1 up, 2 in 2024-06-15T00:03:57.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:56 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:03:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:56 smithi136 ceph-mon[35883]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:56 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2797491966' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033f1a9-211c-4c35-a2b9-33361d67ddd2"}]: dispatch 2024-06-15T00:03:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:56 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2797491966' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a033f1a9-211c-4c35-a2b9-33361d67ddd2"}]': finished 2024-06-15T00:03:57.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:56 smithi136 ceph-mon[35883]: osdmap e11: 2 total, 1 up, 2 in 2024-06-15T00:03:57.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:56 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:03:58.011 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:57 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1273201249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:03:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:57 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1273201249' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:03:59.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:59.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:59.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:59.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:59.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:59.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:03:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:59.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:03:59.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:59.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:03:59.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:03:59.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:03:59.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:03:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:01.090 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:00 smithi063 ceph-mon[28882]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:01.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:00 smithi136 ceph-mon[35883]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:02.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:02.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:03.135 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:02 smithi063 ceph-mon[28882]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:03.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:02 smithi136 ceph-mon[35883]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:04.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-15T00:04:04.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:03 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:03 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-15T00:04:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:03 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:05.166 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:04 smithi063 ceph-mon[28882]: Deploying daemon osd.1 on smithi063 2024-06-15T00:04:05.166 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:04 smithi063 ceph-mon[28882]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:05.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:04 smithi136 ceph-mon[35883]: Deploying daemon osd.1 on smithi063 2024-06-15T00:04:05.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:04 smithi136 ceph-mon[35883]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:07.294 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:06 smithi063 ceph-mon[28882]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:06 smithi136 ceph-mon[35883]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:09.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:09 smithi136 ceph-mon[35883]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:09.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:09.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:09 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:09.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:09 smithi063 ceph-mon[28882]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:09.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:09.399 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:09 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:10.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:11.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:11 smithi136 ceph-mon[35883]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:11.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:11.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:11.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:11 smithi063 ceph-mon[28882]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:11.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:11.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:12.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.436+0000 7ff18b7fe640 1 -- 172.21.15.63:0/745528781 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff1941007f0 con 0x7ff1800653a0 2024-06-15T00:04:12.435 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 1 on host 'smithi063' 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.438+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 msgr2=0x7ff180067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.438+0000 7ff199094640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 0x7ff180067850 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff17c000c00 tx=0x7ff17c035040 comp rx=0 tx=0).stop 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 msgr2=0x7ff19410fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff18400c790 tx=0x7ff1840043c0 comp rx=0 tx=0).stop 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 shutdown_connections 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff1800653a0 0x7ff180067850 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1940fc120 0x7ff1941120b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 --2- 172.21.15.63:0/745528781 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff1940fb7d0 0x7ff19410fb60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:12.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 >> 172.21.15.63:0/745528781 conn(0x7ff1940f6100 msgr2=0x7ff1940f7bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:04:12.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 shutdown_connections 2024-06-15T00:04:12.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:12.439+0000 7ff199094640 1 -- 172.21.15.63:0/745528781 wait complete. 2024-06-15T00:04:13.031 DEBUG:teuthology.orchestra.run.smithi063:osd.1> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.1.service 2024-06-15T00:04:13.034 INFO:tasks.cephadm:Deploying osd.2 on smithi063 with /dev/vg_nvme/lv_2... 2024-06-15T00:04:13.034 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-15T00:04:13.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:13 smithi136 ceph-mon[35883]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:13.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:13 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:13.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:13 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:13.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:13 smithi136 ceph-mon[35883]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-15T00:04:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:13 smithi063 ceph-mon[28882]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:13 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:13 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:13 smithi063 ceph-mon[28882]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-15T00:04:13.477 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:04:14.548 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-mon[28882]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-15T00:04:14.548 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-mon[28882]: osdmap e12: 2 total, 1 up, 2 in 2024-06-15T00:04:14.548 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:14.548 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-mon[28882]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:04:14.548 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-mon[28882]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:14.548 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1[53893]: 2024-06-15T00:04:14.446+0000 7f5ebdeba640 -1 osd.1 0 waiting for initial osdmap 2024-06-15T00:04:14.548 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:14 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1[53893]: 2024-06-15T00:04:14.451+0000 7f5eb74ee640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:04:14.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:14 smithi136 ceph-mon[35883]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-15T00:04:14.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:14 smithi136 ceph-mon[35883]: osdmap e12: 2 total, 1 up, 2 in 2024-06-15T00:04:14.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:14 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:14.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:14 smithi136 ceph-mon[35883]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:04:14.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:14 smithi136 ceph-mon[35883]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:04:15.727 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:15 smithi063 ceph-mon[28882]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:04:15.728 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:15 smithi063 ceph-mon[28882]: osdmap e13: 2 total, 1 up, 2 in 2024-06-15T00:04:15.728 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:15 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:15.728 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:15 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:15.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:15 smithi136 ceph-mon[35883]: from='osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:04:15.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:15 smithi136 ceph-mon[35883]: osdmap e13: 2 total, 1 up, 2 in 2024-06-15T00:04:15.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:15 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:15.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:15 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:16.441 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-15T00:04:16.532 INFO:teuthology.orchestra.run.smithi063.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-15T00:04:16.547 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10+0 records in 2024-06-15T00:04:16.547 INFO:teuthology.orchestra.run.smithi063.stderr:10+0 records out 2024-06-15T00:04:16.547 INFO:teuthology.orchestra.run.smithi063.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127377 s, 823 MB/s 2024-06-15T00:04:16.547 INFO:teuthology.orchestra.run.smithi063.stderr:--> Zapping successful for: 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] boot 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: osdmap e14: 2 total, 2 up, 2 in 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:16.762 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:16 smithi063 ceph-mon[28882]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:16.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:04:16.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:04:16.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: osd.1 [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] boot 2024-06-15T00:04:16.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: osdmap e14: 2 total, 2 up, 2 in 2024-06-15T00:04:16.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:04:16.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:16 smithi136 ceph-mon[35883]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:17.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:17 smithi063 ceph-mon[28882]: osdmap e15: 2 total, 2 up, 2 in 2024-06-15T00:04:17.471 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:17.473 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi063:vg_nvme/lv_2 2024-06-15T00:04:17.754 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:04:17.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:17 smithi136 ceph-mon[35883]: osdmap e15: 2 total, 2 up, 2 in 2024-06-15T00:04:17.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:18.760 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:18 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:18.760 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:18 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:18.760 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:18 smithi063 ceph-mon[28882]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:18.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:18 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:18.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:18 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:18.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:18 smithi136 ceph-mon[35883]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:20.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 -- 172.21.15.63:0/2155102528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 msgr2=0x7fda100fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:20.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 --2- 172.21.15.63:0/2155102528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda100fc6d0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fda000048d0 tx=0x7fda00009da0 comp rx=0 tx=0).stop 2024-06-15T00:04:20.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 -- 172.21.15.63:0/2155102528 shutdown_connections 2024-06-15T00:04:20.447 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 --2- 172.21.15.63:0/2155102528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 0x7fda100fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 --2- 172.21.15.63:0/2155102528 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda100fc6d0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 -- 172.21.15.63:0/2155102528 >> 172.21.15.63:0/2155102528 conn(0x7fda100f7aa0 msgr2=0x7fda100f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 -- 172.21.15.63:0/2155102528 shutdown_connections 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.439+0000 7fda14daa640 1 -- 172.21.15.63:0/2155102528 wait complete. 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 Processor -- start 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 -- start start 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 0x7fda1010d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1010d9b0 con 0x7fda100fc2c0 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda14daa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1010e0f0 con 0x7fda100fd590 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:20.448 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38038/0 (socket says 172.21.15.63:38038) 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 -- 172.21.15.63:0/1023742812 learned_addr learned my addr 172.21.15.63:0/1023742812 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda07fff640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 0x7fda1010d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 -- 172.21.15.63:0/1023742812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 msgr2=0x7fda1010d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 0x7fda1010d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 -- 172.21.15.63:0/1023742812 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda00005040 con 0x7fda100fc2c0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.440+0000 7fda0e59c640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fda0002ccc0 tx=0x7fda00009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.441+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda00038070 con 0x7fda100fc2c0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.441+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1010e310 con 0x7fda100fc2c0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.441+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda0002cea0 con 0x7fda100fc2c0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.441+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda00032df0 con 0x7fda100fc2c0 2024-06-15T00:04:20.449 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.441+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1010e850 con 0x7fda100fc2c0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.442+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda100fb720 con 0x7fda100fc2c0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.442+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fda00030020 con 0x7fda100fc2c0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.443+0000 7fda06ffd640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 0x7fd9fc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.443+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fda000a20f0 con 0x7fda100fc2c0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.443+0000 7fda07fff640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 0x7fd9fc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.443+0000 7fda07fff640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 0x7fd9fc067850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009fd0 tx=0x7fd9f8009e50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:04:20.450 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.447+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fda00070f30 con 0x7fda100fc2c0 2024-06-15T00:04:20.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:20.630+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fda1010eca0 con 0x7fd9fc0653a0 2024-06-15T00:04:21.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:20 smithi063 ceph-mon[28882]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:21.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:20 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:04:21.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:20 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:04:21.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:20 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:21.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:20 smithi136 ceph-mon[35883]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:21.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:20 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:04:21.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:20 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:04:21.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:20 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:22.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:21 smithi063 ceph-mon[28882]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:04:22.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:21 smithi136 ceph-mon[35883]: from='client.14284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi063:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:04:23.111 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:22 smithi063 ceph-mon[28882]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:23.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:22 smithi136 ceph-mon[35883]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:24.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi063 2024-06-15T00:04:24.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:24.037 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi063 to 6432M 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.038 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:23 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:24.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi063 2024-06-15T00:04:24.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi063 to 6432M 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:24.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:23 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1655517002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75d5673-9f50-4b65-8302-911b31b51007"}]: dispatch 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1655517002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a75d5673-9f50-4b65-8302-911b31b51007"}]': finished 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: osdmap e16: 3 total, 2 up, 3 in 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:25.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:24 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2374992999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:04:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1655517002' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a75d5673-9f50-4b65-8302-911b31b51007"}]: dispatch 2024-06-15T00:04:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1655517002' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a75d5673-9f50-4b65-8302-911b31b51007"}]': finished 2024-06-15T00:04:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: osdmap e16: 3 total, 2 up, 3 in 2024-06-15T00:04:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:25.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2374992999' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:04:27.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:26 smithi063 ceph-mon[28882]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:27.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:26 smithi136 ceph-mon[35883]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.672 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:28 smithi063 ceph-mon[28882]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:28.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:28.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:28.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:28.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:28 smithi136 ceph-mon[35883]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:31.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:30 smithi063 ceph-mon[28882]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:31.193 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:30 smithi136 ceph-mon[35883]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:32.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-15T00:04:32.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:32.062 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:32.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-15T00:04:32.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:32.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:33.015 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:32 smithi063 ceph-mon[28882]: Deploying daemon osd.2 on smithi063 2024-06-15T00:04:33.015 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:32 smithi063 ceph-mon[28882]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:33.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:32 smithi136 ceph-mon[35883]: Deploying daemon osd.2 on smithi063 2024-06-15T00:04:33.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:32 smithi136 ceph-mon[35883]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:35.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:34 smithi063 ceph-mon[28882]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:35.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:34 smithi136 ceph-mon[35883]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:37.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:36 smithi136 ceph-mon[35883]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:37.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:37.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:37.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:37.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:36 smithi063 ceph-mon[28882]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:37.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:37.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:37.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:39.039 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:38 smithi063 ceph-mon[28882]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:38 smithi136 ceph-mon[35883]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:40.100 INFO:teuthology.orchestra.run.smithi063.stdout:Created osd(s) 2 on host 'smithi063' 2024-06-15T00:04:40.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.102+0000 7fda06ffd640 1 -- 172.21.15.63:0/1023742812 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fda1010eca0 con 0x7fd9fc0653a0 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.104+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 msgr2=0x7fd9fc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.104+0000 7fda14daa640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 0x7fd9fc067850 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fd9f8009fd0 tx=0x7fd9f8009e50 comp rx=0 tx=0).stop 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 msgr2=0x7fda1010cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fda0002ccc0 tx=0x7fda00009da0 comp rx=0 tx=0).stop 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 shutdown_connections 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd9fc0653a0 0x7fd9fc067850 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda100fd590 0x7fda1010d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 --2- 172.21.15.63:0/1023742812 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fda100fc2c0 0x7fda1010cdf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:40.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 >> 172.21.15.63:0/1023742812 conn(0x7fda100f7aa0 msgr2=0x7fda100f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:04:40.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 shutdown_connections 2024-06-15T00:04:40.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:04:40.105+0000 7fda14daa640 1 -- 172.21.15.63:0/1023742812 wait complete. 2024-06-15T00:04:40.675 DEBUG:teuthology.orchestra.run.smithi063:osd.2> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.2.service 2024-06-15T00:04:40.677 INFO:tasks.cephadm:Deploying osd.3 on smithi136 with /dev/vg_nvme/lv_4... 2024-06-15T00:04:40.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-15T00:04:40.932 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:04:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:41 smithi136 ceph-mon[35883]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:41 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:41 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:41 smithi136 ceph-mon[35883]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-15T00:04:41.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:41 smithi063 ceph-mon[28882]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:41.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:41.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:41 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:41.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:41 smithi063 ceph-mon[28882]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-15T00:04:42.294 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:42 smithi136 ceph-mon[35883]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-15T00:04:42.295 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:42 smithi136 ceph-mon[35883]: osdmap e17: 3 total, 2 up, 3 in 2024-06-15T00:04:42.295 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:42 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:42.295 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:42 smithi136 ceph-mon[35883]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:04:42.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-mon[28882]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-15T00:04:42.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-mon[28882]: osdmap e17: 3 total, 2 up, 3 in 2024-06-15T00:04:42.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:42.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-mon[28882]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]: dispatch 2024-06-15T00:04:42.397 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2[61568]: 2024-06-15T00:04:42.111+0000 7f1e8cc7b640 -1 osd.2 0 waiting for initial osdmap 2024-06-15T00:04:42.397 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:42 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2[61568]: 2024-06-15T00:04:42.117+0000 7f1e892b5640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:04:42.718 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-15T00:04:42.812 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-15T00:04:42.823 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-06-15T00:04:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-06-15T00:04:42.823 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00871296 s, 1.2 GB/s 2024-06-15T00:04:42.823 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: osdmap e18: 3 total, 2 up, 3 in 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:43.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:43 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:43.481 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:04:43.481 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: from='osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi063", "root=default"]}]': finished 2024-06-15T00:04:43.482 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: osdmap e18: 3 total, 2 up, 3 in 2024-06-15T00:04:43.482 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:43.482 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:43.482 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:43.482 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:43 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:43.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_4 2024-06-15T00:04:43.757 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:04:44.421 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:44 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:04:44.421 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:44 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:04:44.421 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:44 smithi063 ceph-mon[28882]: osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] boot 2024-06-15T00:04:44.421 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:44 smithi063 ceph-mon[28882]: osdmap e19: 3 total, 3 up, 3 in 2024-06-15T00:04:44.421 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:44 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:44.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:44 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:04:44.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:44 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:04:44.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:44 smithi136 ceph-mon[35883]: osd.2 [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] boot 2024-06-15T00:04:44.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:44 smithi136 ceph-mon[35883]: osdmap e19: 3 total, 3 up, 3 in 2024-06-15T00:04:44.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:44 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- 172.21.15.136:0/259526405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 msgr2=0x7fb7040feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/259526405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb7040feea0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fb6f40048d0 tx=0x7fb6f4009da0 comp rx=0 tx=0).stop 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- 172.21.15.136:0/259526405 shutdown_connections 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/259526405 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 0x7fb7041017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/259526405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb7040feea0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- 172.21.15.136:0/259526405 >> 172.21.15.136:0/259526405 conn(0x7fb7040f6970 msgr2=0x7fb7040f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- 172.21.15.136:0/259526405 shutdown_connections 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- 172.21.15.136:0/259526405 wait complete. 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 Processor -- start 2024-06-15T00:04:45.267 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.265+0000 7fb70a6c4640 1 -- start start 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 0x7fb70410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70410da40 con 0x7fb7040ff3e0 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb70410e130 con 0x7fb7040659a0 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:59588/0 (socket says 172.21.15.136:59588) 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 -- 172.21.15.136:0/2865842174 learned_addr learned my addr 172.21.15.136:0/2865842174 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb6fbfff640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 0x7fb70410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 -- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 msgr2=0x7fb70410d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:04:45.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 0x7fb70410d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 -- 172.21.15.136:0/2865842174 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f4005040 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb703fff640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009af0 tx=0x7fb6f4009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f4038070 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6f40096f0 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70410e360 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb6f4032da0 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.266+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70410e8a0 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.267+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70404b480 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.268+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 18) v1 ==== 83689+0+0 (secure 0 0 0) 0x7fb6f4030020 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.268+0000 7fb7017fa640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 0x7fb6f0067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.268+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3217+0+0 (secure 0 0 0) 0x7fb6f40a23c0 con 0x7fb7040659a0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.268+0000 7fb6fbfff640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 0x7fb6f0067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:04:45.269 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.269+0000 7fb6fbfff640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 0x7fb6f0067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec004eb0 tx=0x7fb6ec00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:04:45.270 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.273+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb6f4070f50 con 0x7fb7040659a0 2024-06-15T00:04:45.442 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:45.444+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb70410eb10 con 0x7fb6f00653a0 2024-06-15T00:04:45.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:45 smithi063 ceph-mon[28882]: pgmap v73: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:45.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-15T00:04:45.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-15T00:04:45.660 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:45 smithi063 ceph-mon[28882]: osdmap e20: 3 total, 3 up, 3 in 2024-06-15T00:04:45.661 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-15T00:04:45.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:45 smithi136 ceph-mon[35883]: pgmap v73: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:45.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-15T00:04:45.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-15T00:04:45.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:45 smithi136 ceph-mon[35883]: osdmap e20: 3 total, 3 up, 3 in 2024-06-15T00:04:45.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-15T00:04:46.679 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66211]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:04:46.679 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66211]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:46.679 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66211]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:46.679 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66211]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:46.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-15T00:04:46.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: osdmap e21: 3 total, 3 up, 3 in 2024-06-15T00:04:46.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:04:46.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi063 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi063 to 4288M 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:46.680 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:46.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.681 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:46.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: osdmap e21: 3 total, 3 up, 3 in 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='client.24135 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: pgmap v76: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi063 2024-06-15T00:04:46.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi063 to 4288M 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.836 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:46.837 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:46.837 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:46.837 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:04:47.178 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66302]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:04:47.179 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66302]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:47.179 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66302]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:47.179 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:04:46 smithi063 sudo[66302]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:47.515 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:47 smithi136 ceph-mon[35883]: osdmap e22: 3 total, 3 up, 3 in 2024-06-15T00:04:47.515 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:47.515 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:47.664 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:47 smithi063 ceph-mon[28882]: osdmap e22: 3 total, 3 up, 3 in 2024-06-15T00:04:47.665 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:47.665 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:04:47.665 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66373]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:04:47.665 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66373]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:47.665 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66373]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:47.665 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66373]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:47.928 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66411]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:04:47.929 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66411]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:47.929 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66411]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:47.929 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:04:47 smithi063 sudo[66411]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:48.536 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66458]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:04:48.536 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66458]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:48.536 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66458]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:48.536 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66458]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:48.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2595299136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]: dispatch 2024-06-15T00:04:48.536 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]: dispatch 2024-06-15T00:04:48.537 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]': finished 2024-06-15T00:04:48.537 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: osdmap e23: 4 total, 3 up, 4 in 2024-06-15T00:04:48.537 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:04:48.537 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:48 smithi063 ceph-mon[28882]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:48.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/2595299136' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]: dispatch 2024-06-15T00:04:48.873 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]: dispatch 2024-06-15T00:04:48.873 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e56ad7be-c9bf-4618-ab5b-f24ff7209410"}]': finished 2024-06-15T00:04:48.873 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: osdmap e23: 4 total, 3 up, 4 in 2024-06-15T00:04:48.873 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:04:48.873 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:48 smithi136 ceph-mon[35883]: pgmap v79: 1 pgs: 1 creating+activating; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:48.928 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66488]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:04:48.929 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66488]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:48.929 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66488]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:48.929 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:04:48 smithi063 sudo[66488]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:49.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66544]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:04:49.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66544]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:49.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66544]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:49.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66544]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:49.902 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66569]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:04:49.903 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:49.903 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66569]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:49.903 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 sudo[66569]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:49.903 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:49 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/3986184710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:04:50.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:49 smithi136 sudo[39728]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:04:50.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:49 smithi136 sudo[39728]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:50.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:49 smithi136 sudo[39728]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:50.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:49 smithi136 sudo[39728]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:50.094 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:04:50.096+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fb6f4071320 con 0x7fb7040659a0 2024-06-15T00:04:50.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 sudo[39916]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:04:50.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 sudo[39916]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:04:50.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 sudo[39916]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:04:50.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 sudo[39916]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:04:50.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/3986184710' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: mgrmap e19: smithi063.dhizav(active, since 2m), standbys: smithi136.frxuxu 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:04:50.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:50.930 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:51.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:04:51.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:04:51.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:04:51.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi063"}]: dispatch 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: mgrmap e19: smithi063.dhizav(active, since 2m), standbys: smithi136.frxuxu 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "mon metadata", "id": "smithi136"}]: dispatch 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: pgmap v80: 1 pgs: 1 creating+activating; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:51.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:51.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:51 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:51.834 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:51 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:51.835 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:51 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:51.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:51 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:51.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:51 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:04:51.929 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:51 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:04:52.849 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:52 smithi136 ceph-mon[35883]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:52.928 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:52 smithi063 ceph-mon[28882]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:55.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:54 smithi063 ceph-mon[28882]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:54 smithi136 ceph-mon[35883]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:56.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:55 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-15T00:04:56.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:55 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:56.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:55 smithi063 ceph-mon[28882]: Deploying daemon osd.3 on smithi136 2024-06-15T00:04:56.297 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:55 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-15T00:04:56.297 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:55 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:04:56.297 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:55 smithi136 ceph-mon[35883]: Deploying daemon osd.3 on smithi136 2024-06-15T00:04:57.195 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:56 smithi136 ceph-mon[35883]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:57.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:56 smithi063 ceph-mon[28882]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:59.302 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:58 smithi136 ceph-mon[35883]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:04:59.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:58 smithi063 ceph-mon[28882]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:00.284 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:59 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:00.284 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:59 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:00.284 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:04:59 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:59 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:59 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:04:59 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:01.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:00 smithi136 ceph-mon[35883]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:00 smithi063 ceph-mon[28882]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:02.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:02.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:03.046 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:02 smithi136 ceph-mon[35883]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:03.087 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.089+0000 7fb7017fa640 1 -- 172.21.15.136:0/2865842174 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb70410eb10 con 0x7fb6f00653a0 2024-06-15T00:05:03.088 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 3 on host 'smithi136' 2024-06-15T00:05:03.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 msgr2=0x7fb6f0067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:03.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 0x7fb6f0067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec004eb0 tx=0x7fb6ec00b040 comp rx=0 tx=0).stop 2024-06-15T00:05:03.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 msgr2=0x7fb70410ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4009af0 tx=0x7fb6f4009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 shutdown_connections 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fb6f00653a0 0x7fb6f0067850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fb7040ff3e0 0x7fb70410d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 --2- 172.21.15.136:0/2865842174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7040659a0 0x7fb70410ceb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 >> 172.21.15.136:0/2865842174 conn(0x7fb7040f6970 msgr2=0x7fb7040f8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 shutdown_connections 2024-06-15T00:05:03.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:03.092+0000 7fb70a6c4640 1 -- 172.21.15.136:0/2865842174 wait complete. 2024-06-15T00:05:03.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:02 smithi063 ceph-mon[28882]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:03.716 DEBUG:teuthology.orchestra.run.smithi136:osd.3> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.3.service 2024-06-15T00:05:03.718 INFO:tasks.cephadm:Deploying osd.4 on smithi136 with /dev/vg_nvme/lv_3... 2024-06-15T00:05:03.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-15T00:05:04.084 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:05:03 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:05:03.811+0000 7f62b48762c0 -1 osd.3 0 log_to_monitors true 2024-06-15T00:05:04.105 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:05:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:04 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:04 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:04 smithi136 ceph-mon[35883]: from='osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:05:04.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:04 smithi136 ceph-mon[35883]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:05:04.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:04 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:04.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:04 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:04.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:04 smithi063 ceph-mon[28882]: from='osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:05:04.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:04 smithi063 ceph-mon[28882]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:05:05.225 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:05:05.104+0000 7f62affe8640 -1 osd.3 0 waiting for initial osdmap 2024-06-15T00:05:05.226 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:05:05.109+0000 7f62a8e1b640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: osdmap e24: 4 total, 3 up, 4 in 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:05.226 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:05 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:05.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: osdmap e24: 4 total, 3 up, 4 in 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:05.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:05 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:06.101 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-15T00:05:06.190 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-15T00:05:06.202 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-06-15T00:05:06.202 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-06-15T00:05:06.202 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00959735 s, 1.1 GB/s 2024-06-15T00:05:06.202 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-06-15T00:05:06.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:06 smithi063 ceph-mon[28882]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:06.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:06 smithi063 ceph-mon[28882]: osdmap e25: 4 total, 3 up, 4 in 2024-06-15T00:05:06.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:06 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:06.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:06 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:06.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:06 smithi136 ceph-mon[35883]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:06.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:06 smithi136 ceph-mon[35883]: osdmap e25: 4 total, 3 up, 4 in 2024-06-15T00:05:06.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:06 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:06.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:06 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:06.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_3 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: pgmap v90: 1 pgs: 1 peering; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] boot 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: osdmap e26: 4 total, 4 up, 4 in 2024-06-15T00:05:07.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:07 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:07.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:05:07.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:05:07.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: pgmap v90: 1 pgs: 1 peering; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:05:07.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:07.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: osd.3 [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] boot 2024-06-15T00:05:07.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: osdmap e26: 4 total, 4 up, 4 in 2024-06-15T00:05:07.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:07 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:05:07.876 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:05:08.313 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:08 smithi136 ceph-mon[35883]: osdmap e27: 4 total, 4 up, 4 in 2024-06-15T00:05:08.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:08 smithi063 ceph-mon[28882]: osdmap e27: 4 total, 4 up, 4 in 2024-06-15T00:05:09.310 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:09 smithi136 ceph-mon[35883]: pgmap v93: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:09.311 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:09 smithi136 ceph-mon[35883]: osdmap e28: 4 total, 4 up, 4 in 2024-06-15T00:05:09.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:09 smithi063 ceph-mon[28882]: pgmap v93: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:09.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:09 smithi063 ceph-mon[28882]: osdmap e28: 4 total, 4 up, 4 in 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 -- 172.21.15.136:0/2518158335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 msgr2=0x7fe6d40fab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 --2- 172.21.15.136:0/2518158335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d40fab10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40048d0 tx=0x7fe6c4009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 -- 172.21.15.136:0/2518158335 shutdown_connections 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 --2- 172.21.15.136:0/2518158335 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 --2- 172.21.15.136:0/2518158335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d40fab10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.087+0000 7fe6d916a640 1 -- 172.21.15.136:0/2518158335 >> 172.21.15.136:0/2518158335 conn(0x7fe6d40f6100 msgr2=0x7fe6d40f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 -- 172.21.15.136:0/2518158335 shutdown_connections 2024-06-15T00:05:10.086 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 -- 172.21.15.136:0/2518158335 wait complete. 2024-06-15T00:05:10.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 Processor -- start 2024-06-15T00:05:10.088 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 -- start start 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d410d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d410e160 con 0x7fe6d40fb050 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.088+0000 7fe6d916a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d410e2d0 con 0x7fe6d40fa700 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:42090/0 (socket says 172.21.15.136:42090) 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 -- 172.21.15.136:0/71644630 learned_addr learned my addr 172.21.15.136:0/71644630 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6cbfff640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d410d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 -- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 msgr2=0x7fe6d410d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d410d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 -- 172.21.15.136:0/71644630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c4005040 con 0x7fe6d40fa700 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6cbfff640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d410d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d2d9d640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40048a0 tx=0x7fe6c4009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:10.089 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6c4038070 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6c402cea0 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d410e550 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe6c402c070 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.089+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d410ea10 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.090+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d40f9da0 con 0x7fe6d40fa700 2024-06-15T00:05:10.090 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.091+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe6c4030020 con 0x7fe6d40fa700 2024-06-15T00:05:10.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.091+0000 7fe6cb7fe640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 0x7fe6c00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:10.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.091+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fe6c40a27a0 con 0x7fe6d40fa700 2024-06-15T00:05:10.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.091+0000 7fe6cbfff640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 0x7fe6c00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:10.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.092+0000 7fe6cbfff640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 0x7fe6c00678e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc006fd0 tx=0x7fe6bc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:10.093 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.096+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe6c4071030 con 0x7fe6d40fa700 2024-06-15T00:05:10.268 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:10.270+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6d410ed90 con 0x7fe6c0065430 2024-06-15T00:05:10.268 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi136 2024-06-15T00:05:10.268 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: osdmap e29: 4 total, 4 up, 4 in 2024-06-15T00:05:10.268 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi136 to 15936M 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.269 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:10 smithi136 ceph-mon[35883]: osdmap e30: 4 total, 4 up, 4 in 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi136 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: osdmap e29: 4 total, 4 up, 4 in 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi136 to 15936M 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:10.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:10.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:10 smithi063 ceph-mon[28882]: osdmap e30: 4 total, 4 up, 4 in 2024-06-15T00:05:11.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:11 smithi063 ceph-mon[28882]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:05:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:05:11.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:11 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:11.505 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:11 smithi136 ceph-mon[35883]: pgmap v96: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:11.506 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:05:11.506 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:05:11.506 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:11 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:12.158 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:12 smithi136 ceph-mon[35883]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:05:12.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:12 smithi063 ceph-mon[28882]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:05:13.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-06-15T00:05:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/125363693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]: dispatch 2024-06-15T00:05:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]: dispatch 2024-06-15T00:05:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]': finished 2024-06-15T00:05:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: osdmap e31: 5 total, 4 up, 5 in 2024-06-15T00:05:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:13 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/125363693' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]: dispatch 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]: dispatch 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0a8b355f-1872-4d76-bda2-b78a235a0dd5"}]': finished 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: osdmap e31: 5 total, 4 up, 5 in 2024-06-15T00:05:13.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:13 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:14.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:14 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2769246720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:05:14.434 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:14 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/2769246720' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:05:15.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:15 smithi063 ceph-mon[28882]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-06-15T00:05:15.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:15 smithi136 ceph-mon[35883]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-06-15T00:05:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:17 smithi063 ceph-mon[28882]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-15T00:05:17.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:17.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:17 smithi136 ceph-mon[35883]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-06-15T00:05:17.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:19.182 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:19 smithi136 ceph-mon[35883]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-15T00:05:19.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:19 smithi063 ceph-mon[28882]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-15T00:05:20.417 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:20 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-15T00:05:20.417 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:20 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:20.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:20 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-15T00:05:20.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:20 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:21.367 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:21 smithi136 ceph-mon[35883]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-15T00:05:21.367 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:21 smithi136 ceph-mon[35883]: Deploying daemon osd.4 on smithi136 2024-06-15T00:05:21.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:21 smithi063 ceph-mon[28882]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-06-15T00:05:21.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:21 smithi063 ceph-mon[28882]: Deploying daemon osd.4 on smithi136 2024-06-15T00:05:23.428 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:23 smithi136 ceph-mon[35883]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:23.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:23 smithi063 ceph-mon[28882]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:25 smithi063 ceph-mon[28882]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:25.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:25 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:25.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:25 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:25.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:25 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:25.516 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:25 smithi136 ceph-mon[35883]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:25.516 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:25 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:25.516 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:25 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:25.516 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:25 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:27.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:27 smithi063 ceph-mon[28882]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:27.439 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:27 smithi136 ceph-mon[35883]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:28.193 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 4 on host 'smithi136' 2024-06-15T00:05:28.194 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.195+0000 7fe6cb7fe640 1 -- 172.21.15.136:0/71644630 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe6d410ed90 con 0x7fe6c0065430 2024-06-15T00:05:28.195 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 msgr2=0x7fe6c00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 0x7fe6c00678e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc006fd0 tx=0x7fe6bc008040 comp rx=0 tx=0).stop 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 msgr2=0x7fe6d410cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe6c40048a0 tx=0x7fe6c4009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 shutdown_connections 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe6c0065430 0x7fe6c00678e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe6d40fb050 0x7fe6d410d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 --2- 172.21.15.136:0/71644630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6d40fa700 0x7fe6d410cef0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 >> 172.21.15.136:0/71644630 conn(0x7fe6d40f6100 msgr2=0x7fe6d40ff9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 shutdown_connections 2024-06-15T00:05:28.196 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:28.198+0000 7fe6d916a640 1 -- 172.21.15.136:0/71644630 wait complete. 2024-06-15T00:05:28.882 DEBUG:teuthology.orchestra.run.smithi136:osd.4> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.4.service 2024-06-15T00:05:28.884 INFO:tasks.cephadm:Deploying osd.5 on smithi136 with /dev/vg_nvme/lv_2... 2024-06-15T00:05:28.884 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-15T00:05:29.136 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:05:29.018+0000 7fa4d20572c0 -1 osd.4 0 log_to_monitors true 2024-06-15T00:05:29.570 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:05:29.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-mon[35883]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:29.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:29.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:29.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-mon[35883]: from='osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:05:29.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:29 smithi136 ceph-mon[35883]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:05:29.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:29 smithi063 ceph-mon[28882]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:29.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:29 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:29.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:29 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:29.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:29 smithi063 ceph-mon[28882]: from='osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:05:29.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:29 smithi063 ceph-mon[28882]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:05:30.334 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:05:30.208+0000 7fa4cd7c9640 -1 osd.4 0 waiting for initial osdmap 2024-06-15T00:05:30.334 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:05:30.214+0000 7fa4c8600640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:05:30.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-mon[35883]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-15T00:05:30.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-mon[35883]: osdmap e32: 5 total, 4 up, 5 in 2024-06-15T00:05:30.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:30.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-mon[35883]: from='osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:30.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:30 smithi136 ceph-mon[35883]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:30.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:30 smithi063 ceph-mon[28882]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-15T00:05:30.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:30 smithi063 ceph-mon[28882]: osdmap e32: 5 total, 4 up, 5 in 2024-06-15T00:05:30.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:30 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:30.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:30 smithi063 ceph-mon[28882]: from='osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:30.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:30 smithi063 ceph-mon[28882]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:31.392 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: osdmap e33: 5 total, 4 up, 5 in 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:31.431 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] boot 2024-06-15T00:05:31.432 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: osdmap e34: 5 total, 5 up, 5 in 2024-06-15T00:05:31.432 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:31.485 INFO:teuthology.orchestra.run.smithi136.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-15T00:05:31.497 INFO:teuthology.orchestra.run.smithi136.stderr: stderr: 10+0 records in 2024-06-15T00:05:31.497 INFO:teuthology.orchestra.run.smithi136.stderr:10+0 records out 2024-06-15T00:05:31.497 INFO:teuthology.orchestra.run.smithi136.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00937307 s, 1.1 GB/s 2024-06-15T00:05:31.497 INFO:teuthology.orchestra.run.smithi136.stderr:--> Zapping successful for: 2024-06-15T00:05:31.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: osdmap e33: 5 total, 4 up, 5 in 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: osd.4 [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] boot 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: osdmap e34: 5 total, 5 up, 5 in 2024-06-15T00:05:31.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:05:32.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi136:vg_nvme/lv_2 2024-06-15T00:05:32.485 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:32 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:05:32.485 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:32 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:05:32.485 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:32.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:32 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:05:32.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:32 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:05:32.679 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:33.473 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:05:33.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:33 smithi136 ceph-mon[35883]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:33.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:33 smithi136 ceph-mon[35883]: osdmap e35: 5 total, 5 up, 5 in 2024-06-15T00:05:33.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:33 smithi063 ceph-mon[28882]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:33.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:33 smithi063 ceph-mon[28882]: osdmap e35: 5 total, 5 up, 5 in 2024-06-15T00:05:34.433 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:34 smithi136 ceph-mon[35883]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:34.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:34 smithi063 ceph-mon[28882]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 -- 172.21.15.136:0/3930000513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 msgr2=0x7f53b00feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3930000513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b00feea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f53a00048d0 tx=0x7f53a0009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 -- 172.21.15.136:0/3930000513 shutdown_connections 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3930000513 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53b00ff3e0 0x7f53b01017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3930000513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b00feea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 -- 172.21.15.136:0/3930000513 >> 172.21.15.136:0/3930000513 conn(0x7f53b00f6970 msgr2=0x7f53b00f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 -- 172.21.15.136:0/3930000513 shutdown_connections 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.755+0000 7f53b55c4640 1 -- 172.21.15.136:0/3930000513 wait complete. 2024-06-15T00:05:35.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 Processor -- start 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 -- start start 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53b00ff3e0 0x7f53b010d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b010e160 con 0x7f53b00ff3e0 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53b55c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53b010e2d0 con 0x7f53b00659a0 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:36276/0 (socket says 172.21.15.136:36276) 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 -- 172.21.15.136:0/3929013720 learned_addr learned my addr 172.21.15.136:0/3929013720 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 -- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53b00ff3e0 msgr2=0x7f53b010d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53b00ff3e0 0x7f53b010d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 -- 172.21.15.136:0/3929013720 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a0005040 con 0x7f53b00659a0 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.756+0000 7f53aeffd640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009af0 tx=0x7f53a0009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:35.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.757+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53a0038070 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.757+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53a00096f0 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.757+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b010e550 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.757+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53a0032db0 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.757+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b010ea10 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.758+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b00fa5f0 con 0x7f53b00659a0 2024-06-15T00:05:35.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.758+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f53a0030020 con 0x7f53b00659a0 2024-06-15T00:05:35.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.759+0000 7f53a77fe640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 0x7f539c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:35.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.759+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f53a00a2ae0 con 0x7f53b00659a0 2024-06-15T00:05:35.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.759+0000 7f53a7fff640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 0x7f539c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:35.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.759+0000 7f53a7fff640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 0x7f539c0678e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53b010cca0 tx=0x7f5398006d20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:35.760 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.763+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f53a0071200 con 0x7f53b00659a0 2024-06-15T00:05:35.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:35.937+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f53b010eca0 con 0x7f539c065430 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi136 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi136 to 7968M 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:36.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:36.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:36.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:36.085 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:35 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi136 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi136 to 7968M 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:05:36.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:35 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:37.011 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:36 smithi136 ceph-mon[35883]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:37.012 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:36 smithi136 ceph-mon[35883]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:05:37.012 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:05:37.012 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:05:37.012 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:36 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:37.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:36 smithi063 ceph-mon[28882]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:37.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:36 smithi063 ceph-mon[28882]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi136:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:05:37.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:05:37.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:05:37.082 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:36 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/2142364937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]: dispatch 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]: dispatch 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]': finished 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: osdmap e36: 6 total, 5 up, 6 in 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:39.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:38 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/1813600392' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:05:39.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:39.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/2142364937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]: dispatch 2024-06-15T00:05:39.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]: dispatch 2024-06-15T00:05:39.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8a606b86-4fc3-4ba6-96ad-77a252da5847"}]': finished 2024-06-15T00:05:39.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: osdmap e36: 6 total, 5 up, 6 in 2024-06-15T00:05:39.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:39.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:38 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/1813600392' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:05:41.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:40 smithi136 ceph-mon[35883]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:41.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:40 smithi063 ceph-mon[28882]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:43.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:42 smithi063 ceph-mon[28882]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:43.307 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:42 smithi136 ceph-mon[35883]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:45.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:44 smithi136 ceph-mon[35883]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:45.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:44 smithi063 ceph-mon[28882]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:46.217 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-15T00:05:46.217 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:45 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:46.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-15T00:05:46.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:45 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:05:47.015 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:46 smithi136 ceph-mon[35883]: Deploying daemon osd.5 on smithi136 2024-06-15T00:05:47.016 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:46 smithi136 ceph-mon[35883]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:47.016 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:46 smithi063 ceph-mon[28882]: Deploying daemon osd.5 on smithi136 2024-06-15T00:05:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:46 smithi063 ceph-mon[28882]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:05:49.084 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:48 smithi136 ceph-mon[35883]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:49.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:48 smithi063 ceph-mon[28882]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:51.272 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:50 smithi136 ceph-mon[35883]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:51.272 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:51.272 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:51.272 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:50 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:51.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:50 smithi063 ceph-mon[28882]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:51.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:05:51.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:51.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:50 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:53.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:52 smithi063 ceph-mon[28882]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:53.184 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:52 smithi136 ceph-mon[35883]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:54.033 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.035+0000 7f53a77fe640 1 -- 172.21.15.136:0/3929013720 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f53b010eca0 con 0x7f539c065430 2024-06-15T00:05:54.033 INFO:teuthology.orchestra.run.smithi136.stdout:Created osd(s) 5 on host 'smithi136' 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 msgr2=0x7f539c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 0x7f539c0678e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f53b010cca0 tx=0x7f5398006d20 comp rx=0 tx=0).stop 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 msgr2=0x7f53b010cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009af0 tx=0x7f53a0009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 shutdown_connections 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f539c065430 0x7f539c0678e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53b00ff3e0 0x7f53b010d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:54.035 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 --2- 172.21.15.136:0/3929013720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53b00659a0 0x7f53b010cef0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:54.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 >> 172.21.15.136:0/3929013720 conn(0x7f53b00f6970 msgr2=0x7f53b00f6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:54.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 shutdown_connections 2024-06-15T00:05:54.036 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:05:54.037+0000 7f53b55c4640 1 -- 172.21.15.136:0/3929013720 wait complete. 2024-06-15T00:05:54.640 DEBUG:teuthology.orchestra.run.smithi136:osd.5> sudo journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.5.service 2024-06-15T00:05:54.683 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-15T00:05:54.684 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-15T00:05:54.911 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:05:54 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:05:54.771+0000 7efd86ba22c0 -1 osd.5 0 log_to_monitors true 2024-06-15T00:05:54.960 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:05:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:55 smithi136 ceph-mon[35883]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:55 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:55 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:55 smithi136 ceph-mon[35883]: from='osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:05:55.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:55 smithi136 ceph-mon[35883]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:05:55.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:55 smithi063 ceph-mon[28882]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:55.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:55 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:55.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:55 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:55.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:55 smithi063 ceph-mon[28882]: from='osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:05:55.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:55 smithi063 ceph-mon[28882]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:05:56.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-15T00:05:56.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: osdmap e37: 6 total, 5 up, 6 in 2024-06-15T00:05:56.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:56.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:56.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:56.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:56.179 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:56 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:56.334 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:05:56.043+0000 7efd82314640 -1 osd.5 0 waiting for initial osdmap 2024-06-15T00:05:56.334 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:05:56.050+0000 7efd7d14b640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:05:56.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-15T00:05:56.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: osdmap e37: 6 total, 5 up, 6 in 2024-06-15T00:05:56.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:56.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:56.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]: dispatch 2024-06-15T00:05:56.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:56.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:56 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:05:56.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 -- 172.21.15.63:0/2606811597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fb010 msgr2=0x7fc8840fd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:56.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/2606811597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fb010 0x7fc8840fd890 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fc8740048d0 tx=0x7fc874009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:56.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 -- 172.21.15.63:0/2606811597 shutdown_connections 2024-06-15T00:05:56.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/2606811597 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fb010 0x7fc8840fd890 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:56.889 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/2606811597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fa6c0 0x7fc8840faad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:56.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 -- 172.21.15.63:0/2606811597 >> 172.21.15.63:0/2606811597 conn(0x7fc8840f6100 msgr2=0x7fc8840f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:56.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 -- 172.21.15.63:0/2606811597 shutdown_connections 2024-06-15T00:05:56.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.887+0000 7fc88bbd1640 1 -- 172.21.15.63:0/2606811597 wait complete. 2024-06-15T00:05:56.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 Processor -- start 2024-06-15T00:05:56.890 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 -- start start 2024-06-15T00:05:56.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:56.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fb010 0x7fc88410d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:56.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88410dbe0 con 0x7fc8840fa6c0 2024-06-15T00:05:56.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88bbd1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88410e400 con 0x7fc8840fb010 2024-06-15T00:05:56.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40256/0 (socket says 172.21.15.63:40256) 2024-06-15T00:05:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 -- 172.21.15.63:0/554747287 learned_addr learned my addr 172.21.15.63:0/554747287 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:05:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88916c640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fb010 0x7fc88410d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 -- 172.21.15.63:0/554747287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fb010 msgr2=0x7fc88410d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:56.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fb010 0x7fc88410d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 -- 172.21.15.63:0/554747287 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc874005040 con 0x7fc8840fa6c0 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.888+0000 7fc88996d640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc880002a50 tx=0x7fc880004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.889+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc880007c30 con 0x7fc8840fa6c0 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.889+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc88000f040 con 0x7fc8840fa6c0 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.889+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88410e690 con 0x7fc8840fa6c0 2024-06-15T00:05:56.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.889+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc88000daf0 con 0x7fc8840fa6c0 2024-06-15T00:05:56.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.889+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88410ec30 con 0x7fc8840fa6c0 2024-06-15T00:05:56.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.890+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840f9da0 con 0x7fc8840fa6c0 2024-06-15T00:05:56.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.890+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc88000dd30 con 0x7fc8840fa6c0 2024-06-15T00:05:56.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.891+0000 7fc87a7fc640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 0x7fc8700678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:56.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.891+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fc880081c30 con 0x7fc8840fa6c0 2024-06-15T00:05:56.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.891+0000 7fc88916c640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 0x7fc8700678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:56.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.891+0000 7fc88916c640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 0x7fc8700678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc88410ce00 tx=0x7fc874035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:56.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:56.895+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc88004cdf0 con 0x7fc8840fa6c0 2024-06-15T00:05:57.067 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.069+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc88404b480 con 0x7fc8840fa6c0 2024-06-15T00:05:57.067 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:57 smithi063 ceph-mon[28882]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:57.068 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:57 smithi063 ceph-mon[28882]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:57.068 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:57 smithi063 ceph-mon[28882]: osdmap e38: 6 total, 5 up, 6 in 2024-06-15T00:05:57.068 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:57 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:57.068 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:57 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:57.068 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.070+0000 7fc87a7fc640 1 -- 172.21.15.63:0/554747287 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fc880088080 con 0x7fc8840fa6c0 2024-06-15T00:05:57.068 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 msgr2=0x7fc8700678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 0x7fc8700678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc88410ce00 tx=0x7fc874035040 comp rx=0 tx=0).stop 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 msgr2=0x7fc88410d050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc880002a50 tx=0x7fc880004600 comp rx=0 tx=0).stop 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 shutdown_connections 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc870065430 0x7fc8700678e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8840fb010 0x7fc88410d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 --2- 172.21.15.63:0/554747287 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8840fa6c0 0x7fc88410d050 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 >> 172.21.15.63:0/554747287 conn(0x7fc8840f6100 msgr2=0x7fc8840f7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 shutdown_connections 2024-06-15T00:05:57.070 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:57.072+0000 7fc88bbd1640 1 -- 172.21.15.63:0/554747287 wait complete. 2024-06-15T00:05:57.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:57 smithi136 ceph-mon[35883]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:57 smithi136 ceph-mon[35883]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi136", "root=default"]}]': finished 2024-06-15T00:05:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:57 smithi136 ceph-mon[35883]: osdmap e38: 6 total, 5 up, 6 in 2024-06-15T00:05:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:57 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:57.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:57 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:57.709 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":39,"num_osds":6,"num_up_osds":6,"osd_up_since":1718409957,"num_in_osds":6,"osd_in_since":1718409938,"num_remapped_pgs":0} 2024-06-15T00:05:57.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:05:57.979 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:05:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: purged_snaps scrub starts 2024-06-15T00:05:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: purged_snaps scrub ok 2024-06-15T00:05:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] boot 2024-06-15T00:05:58.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: osdmap e39: 6 total, 6 up, 6 in 2024-06-15T00:05:58.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:58.335 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:58 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/554747287' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:05:58.337 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: purged_snaps scrub starts 2024-06-15T00:05:58.337 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: purged_snaps scrub ok 2024-06-15T00:05:58.337 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:58.337 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: osd.5 [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] boot 2024-06-15T00:05:58.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: osdmap e39: 6 total, 6 up, 6 in 2024-06-15T00:05:58.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:05:58.338 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:58 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/554747287' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:05:59.149 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:59 smithi136 ceph-mon[35883]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:59.149 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:05:59 smithi136 ceph-mon[35883]: osdmap e40: 6 total, 6 up, 6 in 2024-06-15T00:05:59.381 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:59 smithi063 ceph-mon[28882]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:05:59.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:05:59 smithi063 ceph-mon[28882]: osdmap e40: 6 total, 6 up, 6 in 2024-06-15T00:05:59.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.567+0000 7f1a7fd04640 1 -- 172.21.15.63:0/2121485212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 msgr2=0x7f1a7806f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.567+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/2121485212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a7806f810 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1a740048d0 tx=0x7f1a74009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 -- 172.21.15.63:0/2121485212 shutdown_connections 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/2121485212 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a780701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/2121485212 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a7806f810 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 -- 172.21.15.63:0/2121485212 >> 172.21.15.63:0/2121485212 conn(0x7f1a780f7e50 msgr2=0x7f1a780fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 -- 172.21.15.63:0/2121485212 shutdown_connections 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 -- 172.21.15.63:0/2121485212 wait complete. 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 Processor -- start 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 -- start start 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.568+0000 7f1a7fd04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7fd04640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:59.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7fd04640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7811a4a0 con 0x7f1a78071440 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7fd04640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a78113900 con 0x7f1a7806fd50 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7d29f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.63:33778/0 (socket says 172.21.15.63:33778) 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7d29f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52024/0 (socket says 172.21.15.63:52024) 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 -- 172.21.15.63:0/285189222 learned_addr learned my addr 172.21.15.63:0/285189222 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 -- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 msgr2=0x7f1a78119e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 -- 172.21.15.63:0/285189222 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74005040 con 0x7f1a7806fd50 2024-06-15T00:05:59.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7d29f640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-15T00:05:59.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7daa0640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a740043d0 tx=0x7f1a74009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a74038070 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a74032d20 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1a7403dc60 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.569+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a78113b80 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.570+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a78114040 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.570+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a78070f50 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.571+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1a74030020 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.571+0000 7f1a6e7fc640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 0x7f1a640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.571+0000 7f1a7d29f640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 0x7f1a640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.572+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1a740a3040 con 0x7f1a7806fd50 2024-06-15T00:05:59.570 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.572+0000 7f1a7d29f640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 0x7f1a640678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a68004fd0 tx=0x7f1a6800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:05:59.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.576+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1a740714f0 con 0x7f1a7806fd50 2024-06-15T00:05:59.749 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.751+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a780fbab0 con 0x7f1a7806fd50 2024-06-15T00:05:59.750 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.752+0000 7f1a6e7fc640 1 -- 172.21.15.63:0/285189222 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11043 (secure 0 0 0) 0x7f1a74074b10 con 0x7f1a7806fd50 2024-06-15T00:05:59.750 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:05:59.750 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","created":"2024-06-15T00:00:30.303091+0000","modified":"2024-06-15T00:05:58.047599+0000","last_up_change":"2024-06-15T00:05:57.041775+0000","last_in_change":"2024-06-15T00:05:38.300602+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:04:44.337397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6803","nonce":24271780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6805","nonce":24271780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6809","nonce":24271780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6807","nonce":24271780}]},"public_addr":"172.21.15.63:6803/24271780","cluster_addr":"172.21.15.63:6805/24271780","heartbeat_back_addr":"172.21.15.63:6809/24271780","heartbeat_front_addr":"172.21.15.63:6807/24271780","state":["exists","up"]},{"osd":1,"uuid":"a033f1a9-211c-4c35-a2b9-33361d67ddd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6811","nonce":387118351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6813","nonce":387118351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6817","nonce":387118351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6815","nonce":387118351}]},"public_addr":"172.21.15.63:6811/387118351","cluster_addr":"172.21.15.63:6813/387118351","heartbeat_back_addr":"172.21.15.63:6817/387118351","heartbeat_front_addr":"172.21.15.63:6815/387118351","state":["exists","up"]},{"osd":2,"uuid":"a75d5673-9f50-4b65-8302-911b31b51007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6819","nonce":1408557675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6821","nonce":1408557675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6825","nonce":1408557675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6823","nonce":1408557675}]},"public_addr":"172.21.15.63:6819/1408557675","cluster_addr":"172.21.15.63:6821/1408557675","heartbeat_back_addr":"172.21.15.63:6825/1408557675","heartbeat_front_addr":"172.21.15.63:6823/1408557675","state":["exists","up"]},{"osd":3,"uuid":"e56ad7be-c9bf-4618-ab5b-f24ff7209410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6801","nonce":1457999668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6803","nonce":1457999668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6807","nonce":1457999668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6805","nonce":1457999668}]},"public_addr":"172.21.15.136:6801/1457999668","cluster_addr":"172.21.15.136:6803/1457999668","heartbeat_back_addr":"172.21.15.136:6807/1457999668","heartbeat_front_addr":"172.21.15.136:6805/1457999668","state":["exists","up"]},{"osd":4,"uuid":"0a8b355f-1872-4d76-bda2-b78a235a0dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6809","nonce":2907351967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6811","nonce":2907351967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6815","nonce":2907351967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6813","nonce":2907351967}]},"public_addr":"172.21.15.136:6809/2907351967","cluster_addr":"172.21.15.136:6811/2907351967","heartbeat_back_addr":"172.21.15.136:6815/2907351967","heartbeat_front_addr":"172.21.15.136:6813/2907351967","state":["exists","up"]},{"osd":5,"uuid":"8a606b86-4fc3-4ba6-96ad-77a252da5847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6817","nonce":554435135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6819","nonce":554435135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6823","nonce":554435135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6821","nonce":554435135}]},"public_addr":"172.21.15.136:6817/554435135","cluster_addr":"172.21.15.136:6819/554435135","heartbeat_back_addr":"172.21.15.136:6823/554435135","heartbeat_front_addr":"172.21.15.136:6821/554435135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:03:45.982289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:13.866416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:41.185534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:04.849041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:30.013186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:55.763468+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2359837402":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2561318253":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/1464602609":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/2944299715":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/3442270887":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2426318916":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6800/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6800/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/3229183344":"2024-06-16T00:01:28.538037+0000","172.21.15.63:6800/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/409167655":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6801/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/647762101":"2024-06-16T00:01:28.538037+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 msgr2=0x7f1a640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 0x7f1a640678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1a68004fd0 tx=0x7f1a6800b040 comp rx=0 tx=0).stop 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 msgr2=0x7f1a78119910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f1a740043d0 tx=0x7f1a74009da0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 shutdown_connections 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1a64065430 0x7f1a640678e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1a78071440 0x7f1a78119e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.752 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 --2- 172.21.15.63:0/285189222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a7806fd50 0x7f1a78119910 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:05:59.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 >> 172.21.15.63:0/285189222 conn(0x7f1a780f7e50 msgr2=0x7f1a780f9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:05:59.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 shutdown_connections 2024-06-15T00:05:59.753 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:05:59.754+0000 7f1a7fd04640 1 -- 172.21.15.63:0/285189222 wait complete. 2024-06-15T00:06:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: Detected new or changed devices on smithi136 2024-06-15T00:06:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: Adjusting osd_memory_target on smithi136 to 5312M 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:06:00.429 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:06:00.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:00 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/285189222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:00.559 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-15T00:04:44.337397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-15T00:06:00.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: Detected new or changed devices on smithi136 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: Adjusting osd_memory_target on smithi136 to 5312M 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:06:00.585 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:00 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/285189222' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:00.824 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:01.260 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:01 smithi063 ceph-mon[28882]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:01.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:01 smithi136 ceph-mon[35883]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- 172.21.15.63:0/2588364514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 msgr2=0x7f34ec0fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/2588364514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec0fe2e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f34e00048d0 tx=0x7f34e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- 172.21.15.63:0/2588364514 shutdown_connections 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/2588364514 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec0fe2e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/2588364514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- 172.21.15.63:0/2588364514 >> 172.21.15.63:0/2588364514 conn(0x7f34ec072110 msgr2=0x7f34ec072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- 172.21.15.63:0/2588364514 shutdown_connections 2024-06-15T00:06:02.342 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- 172.21.15.63:0/2588364514 wait complete. 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 Processor -- start 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.336+0000 7f34f2b0f640 1 -- start start 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f2b0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f2b0f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f2b0f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec113b30 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f2b0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34ec10cf20 con 0x7f34ec0650a0 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52038/0 (socket says 172.21.15.63:52038) 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 -- 172.21.15.63:0/356017032 learned_addr learned my addr 172.21.15.63:0/356017032 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f08ab640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 -- 172.21.15.63:0/356017032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 msgr2=0x7f34ec112f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec112f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.343 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 -- 172.21.15.63:0/356017032 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e0005040 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34f08ab640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.337+0000 7f34ebfff640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.338+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e0038070 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.338+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34e00096f0 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.338+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34ec10d1a0 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.338+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f34e0032da0 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.338+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34ec10d660 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.339+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34ec0fda50 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.339+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f34e0030020 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.340+0000 7f34e97fa640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 0x7f34d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.340+0000 7f34f08ab640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 0x7f34d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.340+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f34e00a2a30 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.340+0000 7f34f08ab640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 0x7f34d40678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009a80 tx=0x7f34dc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:02.344 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.344+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f34e0070f60 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:02.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.519+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34ec10e610 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.519+0000 7f34e97fa640 1 -- 172.21.15.63:0/356017032 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34e0074580 con 0x7f34ec0fdeb0 2024-06-15T00:06:02.517 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2024-06-15T00:06:02.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 msgr2=0x7f34d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 0x7f34d40678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009a80 tx=0x7f34dc008040 comp rx=0 tx=0).stop 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 msgr2=0x7f34ec1134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34e0000c00 tx=0x7f34e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 shutdown_connections 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f34d4065430 0x7f34d40678e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f34ec0fdeb0 0x7f34ec1134a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 --2- 172.21.15.63:0/356017032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34ec0650a0 0x7f34ec112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 >> 172.21.15.63:0/356017032 conn(0x7f34ec072110 msgr2=0x7f34ec100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 shutdown_connections 2024-06-15T00:06:02.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:02.521+0000 7f34f2b0f640 1 -- 172.21.15.63:0/356017032 wait complete. 2024-06-15T00:06:02.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:03.139 INFO:tasks.cephadm:Setting up client nodes... 2024-06-15T00:06:03.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-15T00:06:03.405 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:03.527 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:03 smithi063 ceph-mon[28882]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:03.528 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:03 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/356017032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-15T00:06:03.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:03 smithi136 ceph-mon[35883]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:03.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:03 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/356017032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-15T00:06:05.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 -- 172.21.15.63:0/988767477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 msgr2=0x7f30240fa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 --2- 172.21.15.63:0/988767477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fa3d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30180048d0 tx=0x7f3018009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 -- 172.21.15.63:0/988767477 shutdown_connections 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 --2- 172.21.15.63:0/988767477 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fa3d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 --2- 172.21.15.63:0/988767477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 -- 172.21.15.63:0/988767477 >> 172.21.15.63:0/988767477 conn(0x7f30240f6100 msgr2=0x7f30240f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 -- 172.21.15.63:0/988767477 shutdown_connections 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.013+0000 7f302ac8e640 1 -- 172.21.15.63:0/988767477 wait complete. 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 Processor -- start 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 -- start start 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:05.023 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240fbb90 con 0x7f30240fe270 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f302ac8e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30240fbcd0 con 0x7f30240fd920 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f3023fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f3023fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52062/0 (socket says 172.21.15.63:52062) 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.014+0000 7f3023fff640 1 -- 172.21.15.63:0/2746523753 learned_addr learned my addr 172.21.15.63:0/2746523753 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3028a2a640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3023fff640 1 -- 172.21.15.63:0/2746523753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 msgr2=0x7f30240fd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3023fff640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3023fff640 1 -- 172.21.15.63:0/2746523753 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3018005040 con 0x7f30240fe270 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3028a2a640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f3023fff640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3018009af0 tx=0x7f3018009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3018038070 con 0x7f30240fe270 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30180096f0 con 0x7f30240fe270 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240fbf50 con 0x7f30240fe270 2024-06-15T00:06:05.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3018032db0 con 0x7f30240fe270 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.015+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3024114de0 con 0x7f30240fe270 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.016+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30240f9dc0 con 0x7f30240fe270 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.017+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3018030020 con 0x7f30240fe270 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.017+0000 7f30217fa640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 0x7f300c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.017+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f30180a2c90 con 0x7f30240fe270 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.017+0000 7f3028a2a640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 0x7f300c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.017+0000 7f3028a2a640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 0x7f300c0678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f30240fcec0 tx=0x7f3014009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:05.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.021+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3018071200 con 0x7f30240fe270 2024-06-15T00:06:05.264 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.266+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f302404b480 con 0x7f30240fe270 2024-06-15T00:06:05.267 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.270+0000 7f30217fa640 1 -- 172.21.15.63:0/2746523753 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3018074820 con 0x7f30240fe270 2024-06-15T00:06:05.267 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2024-06-15T00:06:05.267 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDt2mxmdS32DxAAWJcvmyMF3sUBA+j8gXyXRQ== 2024-06-15T00:06:05.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.271+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 msgr2=0x7f300c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:05.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.271+0000 7f302ac8e640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 0x7f300c0678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f30240fcec0 tx=0x7f3014009040 comp rx=0 tx=0).stop 2024-06-15T00:06:05.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 msgr2=0x7f30240fb600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:05.269 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3018009af0 tx=0x7f3018009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 shutdown_connections 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f300c065430 0x7f300c0678e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f30240fe270 0x7f30240fb600 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 --2- 172.21.15.63:0/2746523753 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f30240fd920 0x7f30240fd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 >> 172.21.15.63:0/2746523753 conn(0x7f30240f6100 msgr2=0x7f30240f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 shutdown_connections 2024-06-15T00:06:05.270 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:05.272+0000 7f302ac8e640 1 -- 172.21.15.63:0/2746523753 wait complete. 2024-06-15T00:06:05.567 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:05 smithi063 ceph-mon[28882]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:05.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:05 smithi136 ceph-mon[35883]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:05.945 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2024-06-15T00:06:05.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-15T00:06:05.945 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-15T00:06:05.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-15T00:06:06.248 INFO:teuthology.orchestra.run.smithi136.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi136/config 2024-06-15T00:06:06.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:06 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2746523753' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:06.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:06 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2746523753' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:06:06.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:06 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2746523753' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:06.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:06 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2746523753' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:06:07.287 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:07 smithi136 ceph-mon[35883]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:07.580 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:07 smithi063 ceph-mon[28882]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 -- 172.21.15.136:0/427255090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 msgr2=0x7fc04c0fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 --2- 172.21.15.136:0/427255090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c0fb5c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc0480048d0 tx=0x7fc048009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 -- 172.21.15.136:0/427255090 shutdown_connections 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 --2- 172.21.15.136:0/427255090 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c0fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 --2- 172.21.15.136:0/427255090 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c0fb5c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 -- 172.21.15.136:0/427255090 >> 172.21.15.136:0/427255090 conn(0x7fc04c0f6970 msgr2=0x7fc04c0f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:07.785 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 -- 172.21.15.136:0/427255090 shutdown_connections 2024-06-15T00:06:07.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 -- 172.21.15.136:0/427255090 wait complete. 2024-06-15T00:06:07.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.787+0000 7fc054631640 1 Processor -- start 2024-06-15T00:06:07.786 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc054631640 1 -- start start 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc054631640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c10f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc054631640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc054631640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c10fbe0 con 0x7fc04c0fb1b0 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc054631640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c110400 con 0x7fc04c0fc480 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:50364/0 (socket says 172.21.15.136:50364) 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 -- 172.21.15.136:0/393660810 learned_addr learned my addr 172.21.15.136:0/393660810 (peer_addr_for_me v2:172.21.15.136:0/0) 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 -- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 msgr2=0x7fc04c10f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc0523cd640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c10f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:07.788 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c10f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 -- 172.21.15.136:0/393660810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc048005040 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.788+0000 7fc051bcc640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00eb70 tx=0x7fc03c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.789+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc03c007c30 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.789+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc03c00f040 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.789+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c110630 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.789+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc03c0094b0 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.789+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c110bd0 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.790+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c0fa610 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.790+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc03c009630 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.790+0000 7fc042ffd640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 0x7fc0380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.790+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc03c082160 con 0x7fc04c0fc480 2024-06-15T00:06:07.789 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.791+0000 7fc0523cd640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 0x7fc0380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:07.790 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.791+0000 7fc0523cd640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 0x7fc0380678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc04800b020 tx=0x7fc048009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:07.792 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:07.794+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc03c050690 con 0x7fc04c0fc480 2024-06-15T00:06:08.019 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.021+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc04c04b480 con 0x7fc04c0fc480 2024-06-15T00:06:08.024 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.027+0000 7fc042ffd640 1 -- 172.21.15.136:0/393660810 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc03c053cb0 con 0x7fc04c0fc480 2024-06-15T00:06:08.025 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-06-15T00:06:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQDw2mxmeRVwARAAKb1xoDZbVyZpwZZMpShtjw== 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 msgr2=0x7fc0380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 0x7fc0380678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc04800b020 tx=0x7fc048009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 msgr2=0x7fc04c10f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc03c00eb70 tx=0x7fc03c00b040 comp rx=0 tx=0).stop 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 shutdown_connections 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc038065430 0x7fc0380678e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c0fc480 0x7fc04c10f590 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 --2- 172.21.15.136:0/393660810 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc04c0fb1b0 0x7fc04c10f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 >> 172.21.15.136:0/393660810 conn(0x7fc04c0f6970 msgr2=0x7fc04c0f8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 shutdown_connections 2024-06-15T00:06:08.027 INFO:teuthology.orchestra.run.smithi136.stderr:2024-06-15T00:06:08.029+0000 7fc054631640 1 -- 172.21.15.136:0/393660810 wait complete. 2024-06-15T00:06:08.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:08 smithi136 ceph-mon[35883]: from='client.? 172.21.15.136:0/393660810' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:08.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:08 smithi136 ceph-mon[35883]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:08.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:08 smithi136 ceph-mon[35883]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:06:08.650 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-06-15T00:06:08.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-15T00:06:08.651 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-15T00:06:08.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:08 smithi063 ceph-mon[28882]: from='client.? 172.21.15.136:0/393660810' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:08.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:08 smithi063 ceph-mon[28882]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:06:08.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:08 smithi063 ceph-mon[28882]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:06:08.687 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-15T00:06:08.687 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-15T00:06:08.687 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-15T00:06:08.963 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:09.354 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:09 smithi063 ceph-mon[28882]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:09.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:09 smithi136 ceph-mon[35883]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:10.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 -- 172.21.15.63:0/4016185845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f160406fd50 msgr2=0x7f16040701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:10.384 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 --2- 172.21.15.63:0/4016185845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f160406fd50 0x7f16040701c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f15f00048d0 tx=0x7f15f0009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 -- 172.21.15.63:0/4016185845 shutdown_connections 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 --2- 172.21.15.63:0/4016185845 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f160406fd50 0x7f16040701c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 --2- 172.21.15.63:0/4016185845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1604071440 0x7f160406f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 -- 172.21.15.63:0/4016185845 >> 172.21.15.63:0/4016185845 conn(0x7f16040f7f00 msgr2=0x7f16040fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 -- 172.21.15.63:0/4016185845 shutdown_connections 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.385+0000 7f160b23b640 1 -- 172.21.15.63:0/4016185845 wait complete. 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 Processor -- start 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 -- start start 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160406fd50 0x7f16041195b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160411a140 con 0x7f1604071440 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.386+0000 7f160b23b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1604113550 con 0x7f160406fd50 2024-06-15T00:06:10.385 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46168/0 (socket says 172.21.15.63:46168) 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 -- 172.21.15.63:0/2350846562 learned_addr learned my addr 172.21.15.63:0/2350846562 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1608fd7640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160406fd50 0x7f16041195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 -- 172.21.15.63:0/2350846562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160406fd50 msgr2=0x7f16041195b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160406fd50 0x7f16041195b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 -- 172.21.15.63:0/2350846562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f0005040 con 0x7f1604071440 2024-06-15T00:06:10.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f1603fff640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f15f00043d0 tx=0x7f15f0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f0038070 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16041137d0 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15f0032d20 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f15f003dc60 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.387+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1604113c90 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.388+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1604070f50 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.389+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f15f0030020 con 0x7f1604071440 2024-06-15T00:06:10.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.390+0000 7f16017fa640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 0x7f15f40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:10.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.390+0000 7f1608fd7640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 0x7f15f40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:10.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.390+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f15f00a30a0 con 0x7f1604071440 2024-06-15T00:06:10.388 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.390+0000 7f1608fd7640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 0x7f15f40678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f15f8004eb0 tx=0x7f15f800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:10.391 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.394+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f15f00714f0 con 0x7f1604071440 2024-06-15T00:06:10.609 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.612+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f16040fbb00 con 0x7f1604071440 2024-06-15T00:06:10.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.616+0000 7f16017fa640 1 -- 172.21.15.63:0/2350846562 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161447 (secure 0 0 0) 0x7f15f0074b10 con 0x7f1604071440 2024-06-15T00:06:10.619 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 msgr2=0x7f15f40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 0x7f15f40678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f15f8004eb0 tx=0x7f15f800b040 comp rx=0 tx=0).stop 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 msgr2=0x7f1604119af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f15f00043d0 tx=0x7f15f0009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 shutdown_connections 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f15f4065430 0x7f15f40678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1604071440 0x7f1604119af0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 --2- 172.21.15.63:0/2350846562 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160406fd50 0x7f16041195b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 >> 172.21.15.63:0/2350846562 conn(0x7f16040f7f00 msgr2=0x7f16040f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 shutdown_connections 2024-06-15T00:06:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:10.619+0000 7f160b23b640 1 -- 172.21.15.63:0/2350846562 wait complete. 2024-06-15T00:06:11.436 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi063.dhizav","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":3568086764},{"type":"v1","addr":"172.21.15.63:6801","nonce":3568086764}]},"active_addr":"172.21.15.63:6801/3568086764","active_change":"2024-06-15T00:02:31.823594+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14234,"name":"smithi136.frxuxu","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.63:8443/","prometheus":"http://172.21.15.63:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":4192889399}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2338604353}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3947701596}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3648045183}]}]} 2024-06-15T00:06:11.440 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-15T00:06:11.440 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-15T00:06:11.440 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:06:11.466 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:11 smithi063 ceph-mon[28882]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:11.466 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:11 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2350846562' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-15T00:06:11.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:11 smithi136 ceph-mon[35883]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:11.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:11 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2350846562' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-15T00:06:11.710 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:13.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 -- 172.21.15.63:0/1925795759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 msgr2=0x7f57b41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1925795759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b41026c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f57a40048d0 tx=0x7f57a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 -- 172.21.15.63:0/1925795759 shutdown_connections 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1925795759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57b4102c90 0x7f57b40fc320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1925795759 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b41026c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 -- 172.21.15.63:0/1925795759 >> 172.21.15.63:0/1925795759 conn(0x7f57b40f7e60 msgr2=0x7f57b40fa270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 -- 172.21.15.63:0/1925795759 shutdown_connections 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.252+0000 7f57b91b4640 1 -- 172.21.15.63:0/1925795759 wait complete. 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 Processor -- start 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 -- start start 2024-06-15T00:06:13.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57b4102c90 0x7f57b41134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b4113b30 con 0x7f57b41022f0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b91b4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57b410cee0 con 0x7f57b4102c90 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46174/0 (socket says 172.21.15.63:46174) 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 -- 172.21.15.63:0/1394500762 learned_addr learned my addr 172.21.15.63:0/1394500762 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 -- 172.21.15.63:0/1394500762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57b4102c90 msgr2=0x7f57b41134a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57b4102c90 0x7f57b41134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 -- 172.21.15.63:0/1394500762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4005040 con 0x7f57b41022f0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.253+0000 7f57b2d9d640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f57a40048a0 tx=0x7f57a4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.254+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a4038070 con 0x7f57b41022f0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.254+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57a4032d30 con 0x7f57b41022f0 2024-06-15T00:06:13.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.254+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b410d160 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.254+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f57a4009680 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.254+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b410d620 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.255+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b40fda60 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.255+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f57a4030020 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.256+0000 7f57ab7fe640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 0x7f57a00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.256+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f57a40a21f0 con 0x7f57b41022f0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.256+0000 7f57b259c640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 0x7f57a00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.256+0000 7f57b259c640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 0x7f57a00678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57b410e820 tx=0x7f579c006d20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:13.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.260+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f57a4070720 con 0x7f57b41022f0 2024-06-15T00:06:13.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:13 smithi063 ceph-mon[28882]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:13.434 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.437+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57b40fbae0 con 0x7f57b41022f0 2024-06-15T00:06:13.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.437+0000 7f57ab7fe640 1 -- 172.21.15.63:0/1394500762 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11043 (secure 0 0 0) 0x7f57a4073d40 con 0x7f57b41022f0 2024-06-15T00:06:13.435 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:13.435 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","created":"2024-06-15T00:00:30.303091+0000","modified":"2024-06-15T00:05:58.047599+0000","last_up_change":"2024-06-15T00:05:57.041775+0000","last_in_change":"2024-06-15T00:05:38.300602+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:04:44.337397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6803","nonce":24271780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6805","nonce":24271780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6809","nonce":24271780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6807","nonce":24271780}]},"public_addr":"172.21.15.63:6803/24271780","cluster_addr":"172.21.15.63:6805/24271780","heartbeat_back_addr":"172.21.15.63:6809/24271780","heartbeat_front_addr":"172.21.15.63:6807/24271780","state":["exists","up"]},{"osd":1,"uuid":"a033f1a9-211c-4c35-a2b9-33361d67ddd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6811","nonce":387118351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6813","nonce":387118351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6817","nonce":387118351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6815","nonce":387118351}]},"public_addr":"172.21.15.63:6811/387118351","cluster_addr":"172.21.15.63:6813/387118351","heartbeat_back_addr":"172.21.15.63:6817/387118351","heartbeat_front_addr":"172.21.15.63:6815/387118351","state":["exists","up"]},{"osd":2,"uuid":"a75d5673-9f50-4b65-8302-911b31b51007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6819","nonce":1408557675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6821","nonce":1408557675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6825","nonce":1408557675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6823","nonce":1408557675}]},"public_addr":"172.21.15.63:6819/1408557675","cluster_addr":"172.21.15.63:6821/1408557675","heartbeat_back_addr":"172.21.15.63:6825/1408557675","heartbeat_front_addr":"172.21.15.63:6823/1408557675","state":["exists","up"]},{"osd":3,"uuid":"e56ad7be-c9bf-4618-ab5b-f24ff7209410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6801","nonce":1457999668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6803","nonce":1457999668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6807","nonce":1457999668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6805","nonce":1457999668}]},"public_addr":"172.21.15.136:6801/1457999668","cluster_addr":"172.21.15.136:6803/1457999668","heartbeat_back_addr":"172.21.15.136:6807/1457999668","heartbeat_front_addr":"172.21.15.136:6805/1457999668","state":["exists","up"]},{"osd":4,"uuid":"0a8b355f-1872-4d76-bda2-b78a235a0dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6809","nonce":2907351967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6811","nonce":2907351967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6815","nonce":2907351967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6813","nonce":2907351967}]},"public_addr":"172.21.15.136:6809/2907351967","cluster_addr":"172.21.15.136:6811/2907351967","heartbeat_back_addr":"172.21.15.136:6815/2907351967","heartbeat_front_addr":"172.21.15.136:6813/2907351967","state":["exists","up"]},{"osd":5,"uuid":"8a606b86-4fc3-4ba6-96ad-77a252da5847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6817","nonce":554435135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6819","nonce":554435135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6823","nonce":554435135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6821","nonce":554435135}]},"public_addr":"172.21.15.136:6817/554435135","cluster_addr":"172.21.15.136:6819/554435135","heartbeat_back_addr":"172.21.15.136:6823/554435135","heartbeat_front_addr":"172.21.15.136:6821/554435135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:03:45.982289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:13.866416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:41.185534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:04.849041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:30.013186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:55.763468+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2359837402":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2561318253":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/1464602609":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/2944299715":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/3442270887":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2426318916":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6800/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6800/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/3229183344":"2024-06-16T00:01:28.538037+0000","172.21.15.63:6800/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/409167655":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6801/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/647762101":"2024-06-16T00:01:28.538037+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 msgr2=0x7f57a00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 0x7f57a00678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f57b410e820 tx=0x7f579c006d20 comp rx=0 tx=0).stop 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 msgr2=0x7f57b4112f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f57a40048a0 tx=0x7f57a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 shutdown_connections 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f57a0065430 0x7f57a00678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57b4102c90 0x7f57b41134a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 --2- 172.21.15.63:0/1394500762 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f57b41022f0 0x7f57b4112f60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:13.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 >> 172.21.15.63:0/1394500762 conn(0x7f57b40f7e60 msgr2=0x7f57b40f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:13.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 shutdown_connections 2024-06-15T00:06:13.438 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:13.439+0000 7f57b91b4640 1 -- 172.21.15.63:0/1394500762 wait complete. 2024-06-15T00:06:13.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:13 smithi136 ceph-mon[35883]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:14.098 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-15T00:06:14.099 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:06:14.403 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:14.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:14 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1394500762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:14.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:14 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1394500762' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:15.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:15 smithi063 ceph-mon[28882]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:15.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:15 smithi136 ceph-mon[35883]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:15.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 -- 172.21.15.63:0/2659080025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc102c50 msgr2=0x7fc8dc0fc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:15.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 --2- 172.21.15.63:0/2659080025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc102c50 0x7fc8dc0fc2e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc8d80048d0 tx=0x7fc8d8009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:15.973 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 -- 172.21.15.63:0/2659080025 shutdown_connections 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 --2- 172.21.15.63:0/2659080025 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc102c50 0x7fc8dc0fc2e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 --2- 172.21.15.63:0/2659080025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc102680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.965+0000 7fc8e4091640 1 -- 172.21.15.63:0/2659080025 >> 172.21.15.63:0/2659080025 conn(0x7fc8dc0f7e80 msgr2=0x7fc8dc0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 -- 172.21.15.63:0/2659080025 shutdown_connections 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 -- 172.21.15.63:0/2659080025 wait complete. 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 Processor -- start 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 -- start start 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc102c50 0x7fc8dc113430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc113a30 con 0x7fc8dc1022b0 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e4091640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10ce70 con 0x7fc8dc102c50 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e1e2d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:15.974 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e1e2d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:46196/0 (socket says 172.21.15.63:46196) 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.966+0000 7fc8e1e2d640 1 -- 172.21.15.63:0/1113235223 learned_addr learned my addr 172.21.15.63:0/1113235223 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e162c640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc102c50 0x7fc8dc113430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e1e2d640 1 -- 172.21.15.63:0/1113235223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc102c50 msgr2=0x7fc8dc113430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e1e2d640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc102c50 0x7fc8dc113430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e1e2d640 1 -- 172.21.15.63:0/1113235223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8005040 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e1e2d640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c830 tx=0x7fc8d0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8d000a850 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8d000ae60 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc10d150 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8d00113a0 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.967+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc10d6f0 con 0x7fc8dc1022b0 2024-06-15T00:06:15.975 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.968+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8dc0fda20 con 0x7fc8dc1022b0 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.968+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc8d0012400 con 0x7fc8dc1022b0 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.969+0000 7fc8ce7fc640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 0x7fc8c80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.969+0000 7fc8e162c640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 0x7fc8c80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.969+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc8d0082620 con 0x7fc8dc1022b0 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.969+0000 7fc8e162c640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 0x7fc8c80678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b020 tx=0x7fc8d8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:15.976 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:15.973+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc8d0050bf0 con 0x7fc8dc1022b0 2024-06-15T00:06:16.144 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.147+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc8dc0fbaa0 con 0x7fc8dc1022b0 2024-06-15T00:06:16.146 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.148+0000 7fc8ce7fc640 1 -- 172.21.15.63:0/1113235223 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11043 (secure 0 0 0) 0x7fc8d0054210 con 0x7fc8dc1022b0 2024-06-15T00:06:16.146 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:16.147 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":40,"fsid":"25f32e20-2aaa-11ef-bc9d-c7b262605968","created":"2024-06-15T00:00:30.303091+0000","modified":"2024-06-15T00:05:58.047599+0000","last_up_change":"2024-06-15T00:05:57.041775+0000","last_in_change":"2024-06-15T00:05:38.300602+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:04:44.337397+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a47f2aa1-6dfa-482b-88a6-627c7bfcfb9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6803","nonce":24271780}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6805","nonce":24271780}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6809","nonce":24271780}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":24271780},{"type":"v1","addr":"172.21.15.63:6807","nonce":24271780}]},"public_addr":"172.21.15.63:6803/24271780","cluster_addr":"172.21.15.63:6805/24271780","heartbeat_back_addr":"172.21.15.63:6809/24271780","heartbeat_front_addr":"172.21.15.63:6807/24271780","state":["exists","up"]},{"osd":1,"uuid":"a033f1a9-211c-4c35-a2b9-33361d67ddd2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6811","nonce":387118351}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6813","nonce":387118351}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6817","nonce":387118351}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":387118351},{"type":"v1","addr":"172.21.15.63:6815","nonce":387118351}]},"public_addr":"172.21.15.63:6811/387118351","cluster_addr":"172.21.15.63:6813/387118351","heartbeat_back_addr":"172.21.15.63:6817/387118351","heartbeat_front_addr":"172.21.15.63:6815/387118351","state":["exists","up"]},{"osd":2,"uuid":"a75d5673-9f50-4b65-8302-911b31b51007","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6819","nonce":1408557675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6821","nonce":1408557675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6825","nonce":1408557675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":1408557675},{"type":"v1","addr":"172.21.15.63:6823","nonce":1408557675}]},"public_addr":"172.21.15.63:6819/1408557675","cluster_addr":"172.21.15.63:6821/1408557675","heartbeat_back_addr":"172.21.15.63:6825/1408557675","heartbeat_front_addr":"172.21.15.63:6823/1408557675","state":["exists","up"]},{"osd":3,"uuid":"e56ad7be-c9bf-4618-ab5b-f24ff7209410","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6801","nonce":1457999668}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6803","nonce":1457999668}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6807","nonce":1457999668}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1457999668},{"type":"v1","addr":"172.21.15.136:6805","nonce":1457999668}]},"public_addr":"172.21.15.136:6801/1457999668","cluster_addr":"172.21.15.136:6803/1457999668","heartbeat_back_addr":"172.21.15.136:6807/1457999668","heartbeat_front_addr":"172.21.15.136:6805/1457999668","state":["exists","up"]},{"osd":4,"uuid":"0a8b355f-1872-4d76-bda2-b78a235a0dd5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6809","nonce":2907351967}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6811","nonce":2907351967}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6815","nonce":2907351967}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":2907351967},{"type":"v1","addr":"172.21.15.136:6813","nonce":2907351967}]},"public_addr":"172.21.15.136:6809/2907351967","cluster_addr":"172.21.15.136:6811/2907351967","heartbeat_back_addr":"172.21.15.136:6815/2907351967","heartbeat_front_addr":"172.21.15.136:6813/2907351967","state":["exists","up"]},{"osd":5,"uuid":"8a606b86-4fc3-4ba6-96ad-77a252da5847","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6817","nonce":554435135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6819","nonce":554435135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6823","nonce":554435135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":554435135},{"type":"v1","addr":"172.21.15.136:6821","nonce":554435135}]},"public_addr":"172.21.15.136:6817/554435135","cluster_addr":"172.21.15.136:6819/554435135","heartbeat_back_addr":"172.21.15.136:6823/554435135","heartbeat_front_addr":"172.21.15.136:6821/554435135","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:03:45.982289+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:13.866416+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:04:41.185534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:04.849041+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:30.013186+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:05:55.763468+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.63:0/2359837402":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2561318253":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/1464602609":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6801/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/2944299715":"2024-06-16T00:02:31.823441+0000","172.21.15.63:0/3442270887":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/2426318916":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6800/3768764057":"2024-06-16T00:02:31.823441+0000","172.21.15.63:6800/1148153327":"2024-06-16T00:00:58.665699+0000","172.21.15.63:0/3229183344":"2024-06-16T00:01:28.538037+0000","172.21.15.63:6800/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/409167655":"2024-06-16T00:00:58.665699+0000","172.21.15.63:6801/55223302":"2024-06-16T00:01:28.538037+0000","172.21.15.63:0/647762101":"2024-06-16T00:01:28.538037+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 msgr2=0x7fc8c80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 0x7fc8c80678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc8d800b020 tx=0x7fc8d8035040 comp rx=0 tx=0).stop 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 msgr2=0x7fc8dc112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000c830 tx=0x7fc8d0004430 comp rx=0 tx=0).stop 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 shutdown_connections 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8c8065430 0x7fc8c80678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8dc102c50 0x7fc8dc113430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 --2- 172.21.15.63:0/1113235223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8dc1022b0 0x7fc8dc112ef0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 >> 172.21.15.63:0/1113235223 conn(0x7fc8dc0f7e80 msgr2=0x7fc8dc0f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 shutdown_connections 2024-06-15T00:06:16.148 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:16.150+0000 7fc8e4091640 1 -- 172.21.15.63:0/1113235223 wait complete. 2024-06-15T00:06:16.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:16 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1113235223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:16.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:16 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1113235223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:06:16.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-15T00:06:16.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-15T00:06:16.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-15T00:06:16.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-15T00:06:16.940 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-15T00:06:16.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-15T00:06:17.254 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:17.266 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:17.506 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:17 smithi063 ceph-mon[28882]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:17.507 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:17 smithi136 ceph-mon[35883]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:17.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:17.874 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:17.925 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:17.926 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:17.935 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:19.527 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:19 smithi063 ceph-mon[28882]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:19.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:19 smithi136 ceph-mon[35883]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:21.409 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:21 smithi063 ceph-mon[28882]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:21.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:21 smithi136 ceph-mon[35883]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:22.530 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:22 smithi063 ceph-mon[28882]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:22.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:22 smithi136 ceph-mon[35883]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 -- 172.21.15.63:0/3144242859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 msgr2=0x7f95f40fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 --2- 172.21.15.63:0/3144242859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f40fb230 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f95e40048d0 tx=0x7f95e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 -- 172.21.15.63:0/3144242859 shutdown_connections 2024-06-15T00:06:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 --2- 172.21.15.63:0/3144242859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f40fb230 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.610 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 --2- 172.21.15.63:0/3144242859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 0x7f95f41016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.610+0000 7f95fc521640 1 -- 172.21.15.63:0/3144242859 >> 172.21.15.63:0/3144242859 conn(0x7f95f40f6d90 msgr2=0x7f95f40f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 -- 172.21.15.63:0/3144242859 shutdown_connections 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 -- 172.21.15.63:0/3144242859 wait complete. 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 Processor -- start 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 -- start start 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 0x7f95f4113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4113ce0 con 0x7f95f4101cb0 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95fc521640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f410d090 con 0x7f95f4101310 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95f9abc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95f9abc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43600/0 (socket says 172.21.15.63:43600) 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.611+0000 7f95f9abc640 1 -- 172.21.15.63:0/1050731233 learned_addr learned my addr 172.21.15.63:0/1050731233 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:22.611 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95fa2bd640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 0x7f95f4113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95f9abc640 1 -- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 msgr2=0x7f95f4113110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95f9abc640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 0x7f95f4113110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95f9abc640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4005040 con 0x7f95f4101cb0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95f9abc640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e4038070 con 0x7f95f4101cb0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95e40096f0 con 0x7f95f4101cb0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f410d310 con 0x7f95f4101cb0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95e4032da0 con 0x7f95f4101cb0 2024-06-15T00:06:22.612 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.612+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f410d7d0 con 0x7f95f4101cb0 2024-06-15T00:06:22.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.613+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95f40fa9f0 con 0x7f95f4101cb0 2024-06-15T00:06:22.613 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.614+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f95e4030020 con 0x7f95f4101cb0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.614+0000 7f95eaffd640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 0x7f95e00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.614+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f95e40a2ac0 con 0x7f95f4101cb0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.614+0000 7f95fa2bd640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 0x7f95e00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95eaffd640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 0x7f95e006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95e006dc70 con 0x7f95e006b1b0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95e40a7220 con 0x7f95f4101cb0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95faabe640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 0x7f95e006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95fa2bd640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 0x7f95e00678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95f0006fd0 tx=0x7f95f0008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.615+0000 7f95faabe640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 0x7f95e006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.614 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.616+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== osd.5 v2:172.21.15.136:6816/554435135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f95e006dc70 con 0x7f95e006b1b0 2024-06-15T00:06:22.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.629+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 --> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95f410de00 con 0x7f95e006b1b0 2024-06-15T00:06:22.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95eaffd640 1 -- 172.21.15.63:0/1050731233 <== osd.5 v2:172.21.15.136:6816/554435135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f95f410de00 con 0x7f95e006b1b0 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 msgr2=0x7f95e006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 0x7f95e006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 msgr2=0x7f95e00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 0x7f95e00678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f95f0006fd0 tx=0x7f95f0008040 comp rx=0 tx=0).stop 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 msgr2=0x7f95f4113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f95e4000c00 tx=0x7f95e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 shutdown_connections 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:6816/554435135,v1:172.21.15.136:6817/554435135] conn(0x7f95e006b1b0 0x7f95e006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f95e0065430 0x7f95e00678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f95f4101cb0 0x7f95f4113650 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 --2- 172.21.15.63:0/1050731233 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95f4101310 0x7f95f4113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 >> 172.21.15.63:0/1050731233 conn(0x7f95f40f6d90 msgr2=0x7f95f40657e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 shutdown_connections 2024-06-15T00:06:22.629 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.630+0000 7f95fc521640 1 -- 172.21.15.63:0/1050731233 wait complete. 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 -- 172.21.15.63:0/3412643998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 msgr2=0x7f6a200fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 --2- 172.21.15.63:0/3412643998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a200fcd80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6a140048d0 tx=0x7f6a14009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 -- 172.21.15.63:0/3412643998 shutdown_connections 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 --2- 172.21.15.63:0/3412643998 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a200fcd80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 --2- 172.21.15.63:0/3412643998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 0x7f6a200fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 -- 172.21.15.63:0/3412643998 >> 172.21.15.63:0/3412643998 conn(0x7f6a20072110 msgr2=0x7f6a20072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 -- 172.21.15.63:0/3412643998 shutdown_connections 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.803+0000 7f6a262cd640 1 -- 172.21.15.63:0/3412643998 wait complete. 2024-06-15T00:06:22.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 Processor -- start 2024-06-15T00:06:22.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 -- start start 2024-06-15T00:06:22.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 0x7f6a20115110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a20115c50 con 0x7f6a200fbe80 2024-06-15T00:06:22.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a262cd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2010f090 con 0x7f6a200fb570 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43626/0 (socket says 172.21.15.63:43626) 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 -- 172.21.15.63:0/1242260158 learned_addr learned my addr 172.21.15.63:0/1242260158 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1ffff640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 0x7f6a20115110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 -- 172.21.15.63:0/1242260158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 msgr2=0x7f6a20115110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 0x7f6a20115110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.804+0000 7f6a1f7fe640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a14005040 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a1f7fe640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a140048a0 tx=0x7f6a14009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a14038070 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2010f310 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a1402cef0 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a14009640 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.805+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2010f850 con 0x7f6a200fbe80 2024-06-15T00:06:22.804 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.806+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6a2010fdf0 con 0x7f6a200fbe80 2024-06-15T00:06:22.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6a14030020 con 0x7f6a200fbe80 2024-06-15T00:06:22.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 0x7f6a080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.805 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6a140a28e0 con 0x7f6a200fbe80 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1ffff640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 0x7f6a080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 0x7f6a0806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6a0806dc70 con 0x7f6a0806b1b0 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a140a7220 con 0x7f6a200fbe80 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.807+0000 7f6a2486a640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 0x7f6a0806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.808+0000 7f6a1ffff640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 0x7f6a080678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a10004ed0 tx=0x7f6a1000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.808+0000 7f6a2486a640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 0x7f6a0806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:22.806 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.809+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== osd.2 v2:172.21.15.63:6818/1408557675 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f6a0806dc70 con 0x7f6a0806b1b0 2024-06-15T00:06:22.819 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 --> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6a201008b0 con 0x7f6a0806b1b0 2024-06-15T00:06:22.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a1cff9640 1 -- 172.21.15.63:0/1242260158 <== osd.2 v2:172.21.15.63:6818/1408557675 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6a201008b0 con 0x7f6a0806b1b0 2024-06-15T00:06:22.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 msgr2=0x7f6a0806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 0x7f6a0806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 msgr2=0x7f6a080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 0x7f6a080678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6a10004ed0 tx=0x7f6a1000b040 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.822+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 msgr2=0x7f6a20115650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6a140048a0 tx=0x7f6a14009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 shutdown_connections 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6818/1408557675,v1:172.21.15.63:6819/1408557675] conn(0x7f6a0806b1b0 0x7f6a0806d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a08065430 0x7f6a080678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a200fbe80 0x7f6a20115650 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 --2- 172.21.15.63:0/1242260158 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a200fb570 0x7f6a20115110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 >> 172.21.15.63:0/1242260158 conn(0x7f6a20072110 msgr2=0x7f6a200ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:22.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 shutdown_connections 2024-06-15T00:06:22.822 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:22.823+0000 7f6a262cd640 1 -- 172.21.15.63:0/1242260158 wait complete. 2024-06-15T00:06:24.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- 172.21.15.63:0/2481421122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 msgr2=0x7f3e9c0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:24.562 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/2481421122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c0fc2f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e840048d0 tx=0x7f3e84009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- 172.21.15.63:0/2481421122 shutdown_connections 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/2481421122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c0fc2f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/2481421122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 0x7f3e9c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- 172.21.15.63:0/2481421122 >> 172.21.15.63:0/2481421122 conn(0x7f3e9c0f7e50 msgr2=0x7f3e9c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- 172.21.15.63:0/2481421122 shutdown_connections 2024-06-15T00:06:24.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- 172.21.15.63:0/2481421122 wait complete. 2024-06-15T00:06:24.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 Processor -- start 2024-06-15T00:06:24.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.565+0000 7f3ea11b3640 1 -- start start 2024-06-15T00:06:24.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3ea11b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 0x7f3e9c06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:24.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3ea11b3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:24.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3ea11b3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c070280 con 0x7f3e9c102c60 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3ea11b3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c0703f0 con 0x7f3e9c1022c0 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43652/0 (socket says 172.21.15.63:43652) 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 -- 172.21.15.63:0/265332108 learned_addr learned my addr 172.21.15.63:0/265332108 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9ad9d640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 0x7f3e9c06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 -- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 msgr2=0x7f3e9c06f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 0x7f3e9c06f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e84005040 con 0x7f3e9c102c60 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e9a59c640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e840048a0 tx=0x7f3e84009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.566+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e84038070 con 0x7f3e9c102c60 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.567+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e84032d30 con 0x7f3e9c102c60 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.567+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c119610 con 0x7f3e9c102c60 2024-06-15T00:06:24.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.567+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e84009680 con 0x7f3e9c102c60 2024-06-15T00:06:24.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.567+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c119990 con 0x7f3e9c102c60 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.568+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3e9c0fda30 con 0x7f3e9c102c60 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.568+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3e84030020 con 0x7f3e9c102c60 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e937fe640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 0x7f3e880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e840a2220 con 0x7f3e9c102c60 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e9ad9d640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 0x7f3e880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e937fe640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 0x7f3e8806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3e8806dc70 con 0x7f3e8806b1b0 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e840a4870 con 0x7f3e9c102c60 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e9ad9d640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 0x7f3e880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009b50 tx=0x7f3e8c009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.569+0000 7f3e9b59e640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 0x7f3e8806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:24.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.570+0000 7f3e9b59e640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 0x7f3e8806d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:24.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.571+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== osd.4 v2:172.21.15.136:6808/2907351967 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3e8806dc70 con 0x7f3e8806b1b0 2024-06-15T00:06:24.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 --> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3e9c0f9840 con 0x7f3e8806b1b0 2024-06-15T00:06:24.583 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3e937fe640 1 -- 172.21.15.63:0/265332108 <== osd.4 v2:172.21.15.136:6808/2907351967 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f3e9c0f9840 con 0x7f3e8806b1b0 2024-06-15T00:06:24.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 msgr2=0x7f3e8806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:24.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 0x7f3e8806d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 msgr2=0x7f3e880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:24.584 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 0x7f3e880678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009b50 tx=0x7f3e8c009120 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 msgr2=0x7f3e9c06fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.586+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e840048a0 tx=0x7f3e84009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 shutdown_connections 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:6808/2907351967,v1:172.21.15.136:6809/2907351967] conn(0x7f3e8806b1b0 0x7f3e8806d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e88065430 0x7f3e880678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e9c102c60 0x7f3e9c06fcb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 --2- 172.21.15.63:0/265332108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e9c1022c0 0x7f3e9c06f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 >> 172.21.15.63:0/265332108 conn(0x7f3e9c0f7e50 msgr2=0x7f3e9c0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 shutdown_connections 2024-06-15T00:06:24.585 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:24.587+0000 7f3ea11b3640 1 -- 172.21.15.63:0/265332108 wait complete. 2024-06-15T00:06:25.240 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:24 smithi063 ceph-mon[28882]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 -- 172.21.15.63:0/1983775199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 msgr2=0x7f0b6c0fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1983775199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c0fd6e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0b60004900 tx=0x7f0b60009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 -- 172.21.15.63:0/1983775199 shutdown_connections 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1983775199 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 0x7f0b6c106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1983775199 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c0fd6e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 -- 172.21.15.63:0/1983775199 >> 172.21.15.63:0/1983775199 conn(0x7f0b6c0f6df0 msgr2=0x7f0b6c0f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.242+0000 7f0b72b09640 1 -- 172.21.15.63:0/1983775199 shutdown_connections 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.243+0000 7f0b72b09640 1 -- 172.21.15.63:0/1983775199 wait complete. 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.243+0000 7f0b72b09640 1 Processor -- start 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.243+0000 7f0b72b09640 1 -- start start 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b72b09640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b72b09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 0x7f0b6c1135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b72b09640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c113c70 con 0x7f0b6c0fd310 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b72b09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c10d020 con 0x7f0b6c0fdcb0 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43666/0 (socket says 172.21.15.63:43666) 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 -- 172.21.15.63:0/1770464230 learned_addr learned my addr 172.21.15.63:0/1770464230 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b6bfff640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 0x7f0b6c1135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 -- 172.21.15.63:0/1770464230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 msgr2=0x7f0b6c1135e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:25.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 0x7f0b6c1135e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60005040 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.244+0000 7f0b708a5640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0b60004960 tx=0x7f0b60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.245+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b60038070 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.245+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b6002ae90 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.245+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c10d2a0 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.245+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0b60009560 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.245+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6c10d760 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.246+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0b6c10dc10 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.246+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0b60030020 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b697fa640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 0x7f0b5c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0b600a20d0 con 0x7f0b6c0fd310 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b6bfff640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 0x7f0b5c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b697fa640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 0x7f0b5c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:25.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b5c06dc70 con 0x7f0b5c06b1b0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b600a46c0 con 0x7f0b6c0fd310 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b710a6640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 0x7f0b5c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b6bfff640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 0x7f0b5c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c10e960 tx=0x7f0b58009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.247+0000 7f0b710a6640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 0x7f0b5c06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.248+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== osd.0 v2:172.21.15.63:6802/24271780 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f0b5c06dc70 con 0x7f0b5c06b1b0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.261+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 --> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b6c066010 con 0x7f0b5c06b1b0 2024-06-15T00:06:25.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.261+0000 7f0b697fa640 1 -- 172.21.15.63:0/1770464230 <== osd.0 v2:172.21.15.63:6802/24271780 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b6c066010 con 0x7f0b5c06b1b0 2024-06-15T00:06:25.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 msgr2=0x7f0b5c06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:25.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 0x7f0b5c06d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 msgr2=0x7f0b5c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:25.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 0x7f0b5c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c10e960 tx=0x7f0b58009120 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 msgr2=0x7f0b6c1130a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0b60004960 tx=0x7f0b60009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 shutdown_connections 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0b5c065430 0x7f0b5c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b6c0fdcb0 0x7f0b6c1135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:6802/24271780,v1:172.21.15.63:6803/24271780] conn(0x7f0b5c06b1b0 0x7f0b5c06d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 --2- 172.21.15.63:0/1770464230 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0b6c0fd310 0x7f0b6c1130a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 >> 172.21.15.63:0/1770464230 conn(0x7f0b6c0f6df0 msgr2=0x7f0b6c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 shutdown_connections 2024-06-15T00:06:25.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:25.262+0000 7f0b72b09640 1 -- 172.21.15.63:0/1770464230 wait complete. 2024-06-15T00:06:25.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:24 smithi136 ceph-mon[35883]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:26.002 INFO:teuthology.orchestra.run.smithi063.stdout:81604378646 2024-06-15T00:06:26.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-15T00:06:27.083 INFO:teuthology.orchestra.run.smithi063.stdout:146028888076 2024-06-15T00:06:27.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-15T00:06:27.304 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:26 smithi063 ceph-mon[28882]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:27.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:26 smithi136 ceph-mon[35883]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- 172.21.15.63:0/726746716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 msgr2=0x7fd4a00fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 --2- 172.21.15.63:0/726746716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a00fc2f0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048d0 tx=0x7fd494009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- 172.21.15.63:0/726746716 shutdown_connections 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 --2- 172.21.15.63:0/726746716 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a00fc2f0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 --2- 172.21.15.63:0/726746716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 0x7fd4a0102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- 172.21.15.63:0/726746716 >> 172.21.15.63:0/726746716 conn(0x7fd4a00f7e50 msgr2=0x7fd4a00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- 172.21.15.63:0/726746716 shutdown_connections 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- 172.21.15.63:0/726746716 wait complete. 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 Processor -- start 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.515+0000 7fd4a7929640 1 -- start start 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a7929640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 0x7fd4a006f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a7929640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a7929640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a00701f0 con 0x7fd4a0102c60 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a7929640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a0070360 con 0x7fd4a01022c0 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43682/0 (socket says 172.21.15.63:43682) 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 -- 172.21.15.63:0/3259558345 learned_addr learned my addr 172.21.15.63:0/3259558345 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a56c5640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 0x7fd4a006f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:27.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 -- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 msgr2=0x7fd4a006f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 0x7fd4a006f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494005040 con 0x7fd4a0102c60 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a4ec4640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048a0 tx=0x7fd494009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd494038070 con 0x7fd4a0102c60 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a0070590 con 0x7fd4a0102c60 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd49402cef0 con 0x7fd4a0102c60 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.516+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd494009640 con 0x7fd4a0102c60 2024-06-15T00:06:27.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.517+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a01198f0 con 0x7fd4a0102c60 2024-06-15T00:06:27.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.517+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4a00fda30 con 0x7fd4a0102c60 2024-06-15T00:06:27.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.518+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd494030020 con 0x7fd4a0102c60 2024-06-15T00:06:27.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd49dffb640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 0x7fd4880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:27.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4940a28e0 con 0x7fd4a0102c60 2024-06-15T00:06:27.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd4a56c5640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 0x7fd4880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd49dffb640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 0x7fd48806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd48806dc70 con 0x7fd48806b1b0 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4940a7220 con 0x7fd4a0102c60 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd4a56c5640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 0x7fd4880678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd490004fd0 tx=0x7fd490009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.519+0000 7fd4a5ec6640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 0x7fd48806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.520+0000 7fd4a5ec6640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 0x7fd48806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:27.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.520+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== osd.3 v2:172.21.15.136:6800/1457999668 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd48806dc70 con 0x7fd48806b1b0 2024-06-15T00:06:27.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.534+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 --> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4a00f9820 con 0x7fd48806b1b0 2024-06-15T00:06:27.532 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd49dffb640 1 -- 172.21.15.63:0/3259558345 <== osd.3 v2:172.21.15.136:6800/1457999668 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd4a00f9820 con 0x7fd48806b1b0 2024-06-15T00:06:27.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 msgr2=0x7fd48806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:27.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 0x7fd48806d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.533 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 msgr2=0x7fd4880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 0x7fd4880678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd490004fd0 tx=0x7fd490009040 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 msgr2=0x7fd4a006fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048a0 tx=0x7fd494009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 shutdown_connections 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:6800/1457999668,v1:172.21.15.136:6801/1457999668] conn(0x7fd48806b1b0 0x7fd48806d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.535+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd488065430 0x7fd4880678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.536+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd4a0102c60 0x7fd4a006fcb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.536+0000 7fd4a7929640 1 --2- 172.21.15.63:0/3259558345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4a01022c0 0x7fd4a006f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:27.534 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.536+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 >> 172.21.15.63:0/3259558345 conn(0x7fd4a00f7e50 msgr2=0x7fd4a00f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:27.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.536+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 shutdown_connections 2024-06-15T00:06:27.535 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:27.536+0000 7fd4a7929640 1 -- 172.21.15.63:0/3259558345 wait complete. 2024-06-15T00:06:28.144 INFO:teuthology.orchestra.run.smithi063.stdout:167503724552 2024-06-15T00:06:28.145 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-15T00:06:28.789 INFO:teuthology.orchestra.run.smithi063.stdout:38654705699 2024-06-15T00:06:28.789 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-15T00:06:29.285 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:28 smithi063 ceph-mon[28882]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:29.309 INFO:teuthology.orchestra.run.smithi063.stdout:111669149716 2024-06-15T00:06:29.309 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-15T00:06:29.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:28 smithi136 ceph-mon[35883]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- 172.21.15.63:0/2847190325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c201022a0 msgr2=0x7f7c20102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:29.513 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 --2- 172.21.15.63:0/2847190325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c201022a0 0x7f7c20102670 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7c100048d0 tx=0x7f7c10009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- 172.21.15.63:0/2847190325 shutdown_connections 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 --2- 172.21.15.63:0/2847190325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c20102c40 0x7f7c200fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 --2- 172.21.15.63:0/2847190325 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c201022a0 0x7f7c20102670 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- 172.21.15.63:0/2847190325 >> 172.21.15.63:0/2847190325 conn(0x7f7c200f7e50 msgr2=0x7f7c200fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- 172.21.15.63:0/2847190325 shutdown_connections 2024-06-15T00:06:29.514 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- 172.21.15.63:0/2847190325 wait complete. 2024-06-15T00:06:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 Processor -- start 2024-06-15T00:06:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.516+0000 7f7c2725f640 1 -- start start 2024-06-15T00:06:29.515 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c2725f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c201022a0 0x7f7c20112f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c2725f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c2725f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20113a60 con 0x7f7c20102c40 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c2725f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2010cea0 con 0x7f7c201022a0 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44592/0 (socket says 172.21.15.63:44592) 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 -- 172.21.15.63:0/630784373 learned_addr learned my addr 172.21.15.63:0/630784373 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c24ffb640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c201022a0 0x7f7c20112f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 -- 172.21.15.63:0/630784373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c201022a0 msgr2=0x7f7c20112f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c201022a0 0x7f7c20112f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.517+0000 7f7c1ffff640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c10005040 con 0x7f7c20102c40 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c1ffff640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c14002a50 tx=0x7f7c14004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:29.516 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c14007c30 con 0x7f7c20102c40 2024-06-15T00:06:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2010d180 con 0x7f7c20102c40 2024-06-15T00:06:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c1400f040 con 0x7f7c20102c40 2024-06-15T00:06:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c1400daf0 con 0x7f7c20102c40 2024-06-15T00:06:29.517 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.518+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c2010d720 con 0x7f7c20102c40 2024-06-15T00:06:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.519+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7c2010dc50 con 0x7f7c20102c40 2024-06-15T00:06:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.519+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7c1400dc90 con 0x7f7c20102c40 2024-06-15T00:06:29.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c1d7fa640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 0x7f7c080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7c14081d50 con 0x7f7c20102c40 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c24ffb640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 0x7f7c080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c1d7fa640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 0x7f7c0806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7c0806dc70 con 0x7f7c0806b1b0 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c140873b0 con 0x7f7c20102c40 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.520+0000 7f7c257fc640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 0x7f7c0806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:29.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.521+0000 7f7c24ffb640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 0x7f7c080678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c10004540 tx=0x7f7c10035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:29.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.521+0000 7f7c257fc640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 0x7f7c0806d5c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:29.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.521+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== osd.1 v2:172.21.15.63:6810/387118351 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7c0806dc70 con 0x7f7c0806b1b0 2024-06-15T00:06:29.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.540+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 --> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7c200f9840 con 0x7f7c0806b1b0 2024-06-15T00:06:29.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.541+0000 7f7c1d7fa640 1 -- 172.21.15.63:0/630784373 <== osd.1 v2:172.21.15.63:6810/387118351 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7c200f9840 con 0x7f7c0806b1b0 2024-06-15T00:06:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.542+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 msgr2=0x7f7c0806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.542+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 0x7f7c0806d5c0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.540 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 msgr2=0x7f7c080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 0x7f7c080678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7c10004540 tx=0x7f7c10035040 comp rx=0 tx=0).stop 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 msgr2=0x7f7c20113460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7c14002a50 tx=0x7f7c14004600 comp rx=0 tx=0).stop 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 shutdown_connections 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6810/387118351,v1:172.21.15.63:6811/387118351] conn(0x7f7c0806b1b0 0x7f7c0806d5c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7c08065430 0x7f7c080678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.541 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7c20102c40 0x7f7c20113460 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 --2- 172.21.15.63:0/630784373 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c201022a0 0x7f7c20112f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:29.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.543+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 >> 172.21.15.63:0/630784373 conn(0x7f7c200f7e50 msgr2=0x7f7c200f8e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:29.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.544+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 shutdown_connections 2024-06-15T00:06:29.542 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:29.544+0000 7f7c2725f640 1 -- 172.21.15.63:0/630784373 wait complete. 2024-06-15T00:06:30.167 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:30.187 INFO:teuthology.orchestra.run.smithi063.stdout:60129542172 2024-06-15T00:06:30.188 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-15T00:06:30.216 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:30.987 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:30 smithi063 ceph-mon[28882]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:31.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:30 smithi136 ceph-mon[35883]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:32.284 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:31 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:32.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:31 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:32.373 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:32.374 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:32.374 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:33.131 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:32 smithi063 ceph-mon[28882]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:33.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:32 smithi136 ceph-mon[35883]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:33.546 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 -- 172.21.15.63:0/1925719536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fdc30 msgr2=0x7f6a70106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 --2- 172.21.15.63:0/1925719536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fdc30 0x7f6a70106300 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6a64004900 tx=0x7f6a64009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 -- 172.21.15.63:0/1925719536 shutdown_connections 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 --2- 172.21.15.63:0/1925719536 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fdc30 0x7f6a70106300 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 --2- 172.21.15.63:0/1925719536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fd290 0x7f6a700fd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 -- 172.21.15.63:0/1925719536 >> 172.21.15.63:0/1925719536 conn(0x7f6a700f6d90 msgr2=0x7f6a700f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 -- 172.21.15.63:0/1925719536 shutdown_connections 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.548+0000 7f6a75342640 1 -- 172.21.15.63:0/1925719536 wait complete. 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.549+0000 7f6a75342640 1 Processor -- start 2024-06-15T00:06:33.547 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.549+0000 7f6a75342640 1 -- start start 2024-06-15T00:06:33.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a75342640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 0x7f6a70112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:33.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a75342640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a75342640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a70113ac0 con 0x7f6a700fd290 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a75342640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7010cf00 con 0x7f6a700fdc30 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6e7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.63:32834/0 (socket says 172.21.15.63:32834) 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6e7fc640 1 -- 172.21.15.63:0/2554934151 learned_addr learned my addr 172.21.15.63:0/2554934151 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6effd640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 0x7f6a70112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6effd640 1 -- 172.21.15.63:0/2554934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 msgr2=0x7f6a701134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6effd640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6effd640 1 -- 172.21.15.63:0/2554934151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a64005040 con 0x7f6a700fd290 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6e7fc640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.550+0000 7f6a6effd640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 0x7f6a70112f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a600027a0 tx=0x7f6a60009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:33.549 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.551+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a60002d30 con 0x7f6a700fd290 2024-06-15T00:06:33.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.551+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a6000f040 con 0x7f6a700fd290 2024-06-15T00:06:33.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.551+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7010d1e0 con 0x7f6a700fd290 2024-06-15T00:06:33.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.551+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6a60009470 con 0x7f6a700fd290 2024-06-15T00:06:33.550 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.551+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7010d700 con 0x7f6a700fd290 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.552+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70065df0 con 0x7f6a700fd290 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.552+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6a60007c30 con 0x7f6a700fd290 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.553+0000 7f6a577fe640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 0x7f6a580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.553+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6a600821b0 con 0x7f6a700fd290 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.553+0000 7f6a6e7fc640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 0x7f6a580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.553+0000 7f6a6e7fc640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 0x7f6a580678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a7010e840 tx=0x7f6a64035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:33.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.557+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6a600506e0 con 0x7f6a700fd290 2024-06-15T00:06:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.741+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f6a7004b480 con 0x7f6a700fd290 2024-06-15T00:06:33.738 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.741+0000 7f6a577fe640 1 -- 172.21.15.63:0/2554934151 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6a60053d00 con 0x7f6a700fd290 2024-06-15T00:06:33.739 INFO:teuthology.orchestra.run.smithi063.stdout:146028888078 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 msgr2=0x7f6a580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 0x7f6a580678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f6a7010e840 tx=0x7f6a64035040 comp rx=0 tx=0).stop 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 msgr2=0x7f6a70112f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 0x7f6a70112f80 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6a600027a0 tx=0x7f6a60009f30 comp rx=0 tx=0).stop 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 shutdown_connections 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6a58065430 0x7f6a580678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a700fdc30 0x7f6a701134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 --2- 172.21.15.63:0/2554934151 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6a700fd290 0x7f6a70112f80 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 >> 172.21.15.63:0/2554934151 conn(0x7f6a700f6d90 msgr2=0x7f6a700f8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:33.741 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 shutdown_connections 2024-06-15T00:06:33.742 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:33.743+0000 7f6a75342640 1 -- 172.21.15.63:0/2554934151 wait complete. 2024-06-15T00:06:34.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:33 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2554934151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-15T00:06:34.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:33 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2554934151' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-15T00:06:35.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:34 smithi063 ceph-mon[28882]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:35.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:34 smithi136 ceph-mon[35883]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:36.248 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888076 got 146028888078 for osd.4 2024-06-15T00:06:36.248 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:36.398 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:36.399 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 -- 172.21.15.63:0/77415532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c101b80 msgr2=0x7f166c0fbaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 --2- 172.21.15.63:0/77415532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c101b80 0x7f166c0fbaf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f16600048d0 tx=0x7f1660009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 -- 172.21.15.63:0/77415532 shutdown_connections 2024-06-15T00:06:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 --2- 172.21.15.63:0/77415532 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c101b80 0x7f166c0fbaf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 --2- 172.21.15.63:0/77415532 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c1011e0 0x7f166c1015b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.402+0000 7f16709c2640 1 -- 172.21.15.63:0/77415532 >> 172.21.15.63:0/77415532 conn(0x7f166c070a40 msgr2=0x7f166c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 -- 172.21.15.63:0/77415532 shutdown_connections 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 -- 172.21.15.63:0/77415532 wait complete. 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 Processor -- start 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 -- start start 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c101b80 0x7f166c10c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c10c770 con 0x7f166c1011e0 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f16709c2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c10c8b0 con 0x7f166c101b80 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.403+0000 7f166a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44626/0 (socket says 172.21.15.63:44626) 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 -- 172.21.15.63:0/379869928 learned_addr learned my addr 172.21.15.63:0/379869928 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f1669d9b640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c101b80 0x7f166c10c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 -- 172.21.15.63:0/379869928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c101b80 msgr2=0x7f166c10c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c101b80 0x7f166c10c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.402 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 -- 172.21.15.63:0/379869928 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1660005040 con 0x7f166c1011e0 2024-06-15T00:06:36.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f166a59c640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f165c002a50 tx=0x7f165c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:36.403 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f165c007c30 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c119360 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f165c00f040 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f165c00da70 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.404+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c119710 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.406+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f165c00dcb0 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.406+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c0fcb00 con 0x7f166c1011e0 2024-06-15T00:06:36.404 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.406+0000 7f165affd640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 0x7f16540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:36.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.406+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f165c081b90 con 0x7f166c1011e0 2024-06-15T00:06:36.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.406+0000 7f1669d9b640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 0x7f16540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:36.405 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.407+0000 7f1669d9b640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 0x7f16540678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f166c0f97a0 tx=0x7f1660035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:36.408 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.411+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f165c04cdf0 con 0x7f166c1011e0 2024-06-15T00:06:36.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.588+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f166c0f9ab0 con 0x7f166c1011e0 2024-06-15T00:06:36.586 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.589+0000 7f165affd640 1 -- 172.21.15.63:0/379869928 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f165c050250 con 0x7f166c1011e0 2024-06-15T00:06:36.587 INFO:teuthology.orchestra.run.smithi063.stdout:81604378649 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 msgr2=0x7f16540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 0x7f16540678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f166c0f97a0 tx=0x7f1660035040 comp rx=0 tx=0).stop 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 msgr2=0x7f166c10bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f165c002a50 tx=0x7f165c004600 comp rx=0 tx=0).stop 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 shutdown_connections 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1654065430 0x7f16540678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f166c101b80 0x7f166c10c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 --2- 172.21.15.63:0/379869928 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f166c1011e0 0x7f166c10bcf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 >> 172.21.15.63:0/379869928 conn(0x7f166c070a40 msgr2=0x7f166c0fedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 shutdown_connections 2024-06-15T00:06:36.589 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:36.591+0000 7f16709c2640 1 -- 172.21.15.63:0/379869928 wait complete. 2024-06-15T00:06:37.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.016+0000 7fc4734e1640 1 -- 172.21.15.63:0/1843035717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 msgr2=0x7fc46c0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.016+0000 7fc4734e1640 1 --2- 172.21.15.63:0/1843035717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c0fcd80 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc4680048d0 tx=0x7fc468009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- 172.21.15.63:0/1843035717 shutdown_connections 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 --2- 172.21.15.63:0/1843035717 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c0fcd80 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 --2- 172.21.15.63:0/1843035717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 0x7fc46c0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- 172.21.15.63:0/1843035717 >> 172.21.15.63:0/1843035717 conn(0x7fc46c072110 msgr2=0x7fc46c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- 172.21.15.63:0/1843035717 shutdown_connections 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- 172.21.15.63:0/1843035717 wait complete. 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 Processor -- start 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- start start 2024-06-15T00:06:37.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 0x7fc46c111e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c112940 con 0x7fc46c0fbe80 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.017+0000 7fc4734e1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46c10bd80 con 0x7fc46c0fb570 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44638/0 (socket says 172.21.15.63:44638) 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 -- 172.21.15.63:0/487708888 learned_addr learned my addr 172.21.15.63:0/487708888 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc47127d640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 0x7fc46c111e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 -- 172.21.15.63:0/487708888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 msgr2=0x7fc46c111e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 0x7fc46c111e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 -- 172.21.15.63:0/487708888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc468005040 con 0x7fc46c0fbe80 2024-06-15T00:06:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc470a7c640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc4680048a0 tx=0x7fc468009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc468038070 con 0x7fc46c0fbe80 2024-06-15T00:06:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc468032d30 con 0x7fc46c0fbe80 2024-06-15T00:06:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc46c10c000 con 0x7fc46c0fbe80 2024-06-15T00:06:37.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.018+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc468009680 con 0x7fc46c0fbe80 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.019+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc46c10c540 con 0x7fc46c0fbe80 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.019+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46c0fc920 con 0x7fc46c0fbe80 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.020+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc468030020 con 0x7fc46c0fbe80 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.020+0000 7fc461ffb640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 0x7fc4580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.021+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc4680a2140 con 0x7fc46c0fbe80 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.021+0000 7fc47127d640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 0x7fc4580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:37.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.021+0000 7fc47127d640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 0x7fc4580678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc45c007fd0 tx=0x7fc45c004330 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:37.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.025+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc468070720 con 0x7fc46c0fbe80 2024-06-15T00:06:37.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:36 smithi063 ceph-mon[28882]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:37.126 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:36 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/379869928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-15T00:06:37.198 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.201+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fc46c10d470 con 0x7fc46c0fbe80 2024-06-15T00:06:37.199 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.202+0000 7fc461ffb640 1 -- 172.21.15.63:0/487708888 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc468073d40 con 0x7fc46c0fbe80 2024-06-15T00:06:37.199 INFO:teuthology.orchestra.run.smithi063.stdout:111669149717 2024-06-15T00:06:37.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 msgr2=0x7fc4580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:37.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 0x7fc4580678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc45c007fd0 tx=0x7fc45c004330 comp rx=0 tx=0).stop 2024-06-15T00:06:37.201 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 msgr2=0x7fc46c112340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc4680048a0 tx=0x7fc468009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 shutdown_connections 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc458065430 0x7fc4580678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc46c0fbe80 0x7fc46c112340 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 --2- 172.21.15.63:0/487708888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc46c0fb570 0x7fc46c111e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 >> 172.21.15.63:0/487708888 conn(0x7fc46c072110 msgr2=0x7fc46c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 shutdown_connections 2024-06-15T00:06:37.202 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:37.204+0000 7fc4734e1640 1 -- 172.21.15.63:0/487708888 wait complete. 2024-06-15T00:06:37.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:36 smithi136 ceph-mon[35883]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:37.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:36 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/379869928' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-15T00:06:38.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:37 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/487708888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-15T00:06:38.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:37 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/487708888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-15T00:06:38.634 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378646 got 81604378649 for osd.2 2024-06-15T00:06:38.634 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:39.235 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:38 smithi136 ceph-mon[35883]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:39.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:38 smithi063 ceph-mon[28882]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:40.112 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149716 got 111669149717 for osd.3 2024-06-15T00:06:40.112 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 -- 172.21.15.63:0/3237701816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 msgr2=0x7fde580fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 --2- 172.21.15.63:0/3237701816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde580fcd80 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fde400048d0 tx=0x7fde40009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 -- 172.21.15.63:0/3237701816 shutdown_connections 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 --2- 172.21.15.63:0/3237701816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde580fcd80 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 --2- 172.21.15.63:0/3237701816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 0x7fde580fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 -- 172.21.15.63:0/3237701816 >> 172.21.15.63:0/3237701816 conn(0x7fde58072110 msgr2=0x7fde58072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 -- 172.21.15.63:0/3237701816 shutdown_connections 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.885+0000 7fde5c97c640 1 -- 172.21.15.63:0/3237701816 wait complete. 2024-06-15T00:06:41.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 Processor -- start 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 -- start start 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 0x7fde5810bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5810c750 con 0x7fde580fbe80 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5c97c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde5810c890 con 0x7fde580fb570 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55524/0 (socket says 172.21.15.63:55524) 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 -- 172.21.15.63:0/1204495517 learned_addr learned my addr 172.21.15.63:0/1204495517 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde5659c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 0x7fde5810bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 -- 172.21.15.63:0/1204495517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 msgr2=0x7fde5810bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 0x7fde5810bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.886+0000 7fde55d9b640 1 -- 172.21.15.63:0/1204495517 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde40005040 con 0x7fde580fbe80 2024-06-15T00:06:41.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde55d9b640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fde400048a0 tx=0x7fde40009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde40038070 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde40032d30 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fde40009680 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde5810cb10 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.887+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde58119610 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.888+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde580fc920 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.888+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fde40030020 con 0x7fde580fbe80 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.889+0000 7fde4effd640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 0x7fde440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.889+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fde400a21c0 con 0x7fde580fbe80 2024-06-15T00:06:41.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.889+0000 7fde5659c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 0x7fde440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.889+0000 7fde5659c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 0x7fde440678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde480041d0 tx=0x7fde48009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:41.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:40.893+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fde400706f0 con 0x7fde580fbe80 2024-06-15T00:06:41.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.074+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fde58072c20 con 0x7fde580fbe80 2024-06-15T00:06:41.076 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.075+0000 7fde4effd640 1 -- 172.21.15.63:0/1204495517 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde40073d10 con 0x7fde580fbe80 2024-06-15T00:06:41.076 INFO:teuthology.orchestra.run.smithi063.stdout:38654705702 2024-06-15T00:06:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 msgr2=0x7fde440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 0x7fde440678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fde480041d0 tx=0x7fde48009040 comp rx=0 tx=0).stop 2024-06-15T00:06:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 msgr2=0x7fde5810c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.077 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fde400048a0 tx=0x7fde40009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 shutdown_connections 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fde44065430 0x7fde440678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fde580fbe80 0x7fde5810c210 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 --2- 172.21.15.63:0/1204495517 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde580fb570 0x7fde5810bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 >> 172.21.15.63:0/1204495517 conn(0x7fde58072110 msgr2=0x7fde580f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 shutdown_connections 2024-06-15T00:06:41.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.077+0000 7fde5c97c640 1 -- 172.21.15.63:0/1204495517 wait complete. 2024-06-15T00:06:41.248 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:40 smithi063 ceph-mon[28882]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 -- 172.21.15.63:0/857921319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 msgr2=0x7fa5200fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 --2- 172.21.15.63:0/857921319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fae30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa5080048d0 tx=0x7fa508009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 -- 172.21.15.63:0/857921319 shutdown_connections 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 --2- 172.21.15.63:0/857921319 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fae30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 --2- 172.21.15.63:0/857921319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa520102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 -- 172.21.15.63:0/857921319 >> 172.21.15.63:0/857921319 conn(0x7fa5200709e0 msgr2=0x7fa520070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:41.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 -- 172.21.15.63:0/857921319 shutdown_connections 2024-06-15T00:06:41.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.255+0000 7fa525816640 1 -- 172.21.15.63:0/857921319 wait complete. 2024-06-15T00:06:41.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 Processor -- start 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 -- start start 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa5200fc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200fc6b0 con 0x7fa5200fa9c0 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa525816640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5200fc820 con 0x7fa520102370 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa51effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa51effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55544/0 (socket says 172.21.15.63:55544) 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa51effd640 1 -- 172.21.15.63:0/1812542648 learned_addr learned my addr 172.21.15.63:0/1812542648 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.256+0000 7fa5167fc640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa5200fc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa51effd640 1 -- 172.21.15.63:0/1812542648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 msgr2=0x7fa5200fc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa51effd640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa5200fc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa51effd640 1 -- 172.21.15.63:0/1812542648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa508005040 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa5167fc640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa5200fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa51effd640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa51000b790 tx=0x7fa5100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa51000c700 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa51000cd10 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5201192e0 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa510013370 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.257+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa520119720 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.258+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520065570 con 0x7fa5200fa9c0 2024-06-15T00:06:41.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.258+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa510012070 con 0x7fa5200fa9c0 2024-06-15T00:06:41.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.259+0000 7fa517fff640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 0x7fa50c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:41.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.259+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa510082020 con 0x7fa5200fa9c0 2024-06-15T00:06:41.258 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.259+0000 7fa5167fc640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 0x7fa50c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:41.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.259+0000 7fa5167fc640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 0x7fa50c0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa508007ac0 tx=0x7fa508035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:41.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.263+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa510050550 con 0x7fa5200fa9c0 2024-06-15T00:06:41.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:40 smithi136 ceph-mon[35883]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:41.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.440+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fa520074980 con 0x7fa5200fa9c0 2024-06-15T00:06:41.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.440+0000 7fa517fff640 1 -- 172.21.15.63:0/1812542648 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa520074980 con 0x7fa5200fa9c0 2024-06-15T00:06:41.438 INFO:teuthology.orchestra.run.smithi063.stdout:167503724556 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 msgr2=0x7fa50c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 0x7fa50c0678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa508007ac0 tx=0x7fa508035040 comp rx=0 tx=0).stop 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 msgr2=0x7fa5200fbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fa51000b790 tx=0x7fa5100091d0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 shutdown_connections 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa50c065430 0x7fa50c0678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa520102370 0x7fa5200fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 --2- 172.21.15.63:0/1812542648 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa5200fa9c0 0x7fa5200fbba0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 >> 172.21.15.63:0/1812542648 conn(0x7fa5200709e0 msgr2=0x7fa5200f9d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 shutdown_connections 2024-06-15T00:06:41.440 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:41.442+0000 7fa525816640 1 -- 172.21.15.63:0/1812542648 wait complete. 2024-06-15T00:06:41.960 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-06-15T00:06:41.960 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:42.303 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:41 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1204495517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-15T00:06:42.303 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:41 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1812542648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-15T00:06:42.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:41 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1204495517' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-15T00:06:42.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:41 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1812542648' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-15T00:06:43.007 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:42 smithi063 ceph-mon[28882]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:43.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724552 got 167503724556 for osd.5 2024-06-15T00:06:43.033 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:43.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 -- 172.21.15.63:0/192285859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d81022c0 msgr2=0x7f05d8102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:43.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/192285859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d81022c0 0x7f05d8102690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f05c80048d0 tx=0x7f05c8009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 -- 172.21.15.63:0/192285859 shutdown_connections 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/192285859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d8102c60 0x7f05d80fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/192285859 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d81022c0 0x7f05d8102690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 -- 172.21.15.63:0/192285859 >> 172.21.15.63:0/192285859 conn(0x7f05d80f7e50 msgr2=0x7f05d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 -- 172.21.15.63:0/192285859 shutdown_connections 2024-06-15T00:06:43.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.254+0000 7f05dd5a7640 1 -- 172.21.15.63:0/192285859 wait complete. 2024-06-15T00:06:43.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 Processor -- start 2024-06-15T00:06:43.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 -- start start 2024-06-15T00:06:43.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d81022c0 0x7f05d8113240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d8113d80 con 0x7f05d8102c60 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05dd5a7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d810d1c0 con 0x7f05d81022c0 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55560/0 (socket says 172.21.15.63:55560) 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 -- 172.21.15.63:0/1380393543 learned_addr learned my addr 172.21.15.63:0/1380393543 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d6ffd640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d81022c0 0x7f05d8113240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 -- 172.21.15.63:0/1380393543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d81022c0 msgr2=0x7f05d8113240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d81022c0 0x7f05d8113240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.255+0000 7f05d67fc640 1 -- 172.21.15.63:0/1380393543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c8005040 con 0x7f05d8102c60 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05d67fc640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05c000c830 tx=0x7f05c0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c000a850 con 0x7f05d8102c60 2024-06-15T00:06:43.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f05c000ae60 con 0x7f05d8102c60 2024-06-15T00:06:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05d810d4a0 con 0x7f05d8102c60 2024-06-15T00:06:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f05c00113a0 con 0x7f05d8102c60 2024-06-15T00:06:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.256+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d810da40 con 0x7f05d8102c60 2024-06-15T00:06:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.257+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d80fda30 con 0x7f05d8102c60 2024-06-15T00:06:43.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.257+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f05c0012400 con 0x7f05d8102c60 2024-06-15T00:06:43.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.258+0000 7f05cf7fe640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 0x7f05c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:43.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.258+0000 7f05d6ffd640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 0x7f05c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:43.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.258+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f05c0082630 con 0x7f05d8102c60 2024-06-15T00:06:43.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.258+0000 7f05d6ffd640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 0x7f05c40678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f05c8004540 tx=0x7f05c8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:43.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.262+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f05c0050c00 con 0x7f05d8102c60 2024-06-15T00:06:43.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:42 smithi136 ceph-mon[35883]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:43.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.437+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f05d80fbab0 con 0x7f05d8102c60 2024-06-15T00:06:43.435 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.437+0000 7f05cf7fe640 1 -- 172.21.15.63:0/1380393543 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f05c0054220 con 0x7f05d8102c60 2024-06-15T00:06:43.435 INFO:teuthology.orchestra.run.smithi063.stdout:60129542175 2024-06-15T00:06:43.436 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 msgr2=0x7f05c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 0x7f05c40678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f05c8004540 tx=0x7f05c8035040 comp rx=0 tx=0).stop 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 msgr2=0x7f05d8113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f05c000c830 tx=0x7f05c0004430 comp rx=0 tx=0).stop 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 shutdown_connections 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f05c4065430 0x7f05c40678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f05d8102c60 0x7f05d8113780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 --2- 172.21.15.63:0/1380393543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05d81022c0 0x7f05d8113240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 >> 172.21.15.63:0/1380393543 conn(0x7f05d80f7e50 msgr2=0x7f05d80f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 shutdown_connections 2024-06-15T00:06:43.437 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:43.439+0000 7f05dd5a7640 1 -- 172.21.15.63:0/1380393543 wait complete. 2024-06-15T00:06:44.120 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:43 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1380393543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-15T00:06:44.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542172 got 60129542175 for osd.1 2024-06-15T00:06:44.145 DEBUG:teuthology.parallel:result is None 2024-06-15T00:06:44.145 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-15T00:06:44.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-15T00:06:44.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:43 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1380393543' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-15T00:06:44.410 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:45.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:44 smithi063 ceph-mon[28882]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:45.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:44 smithi136 ceph-mon[35883]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- 172.21.15.63:0/1947317375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 msgr2=0x7f57680fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 --2- 172.21.15.63:0/1947317375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f57680fc2d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f57500048d0 tx=0x7f5750009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- 172.21.15.63:0/1947317375 shutdown_connections 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 --2- 172.21.15.63:0/1947317375 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f57680fc2d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 --2- 172.21.15.63:0/1947317375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f5768102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- 172.21.15.63:0/1947317375 >> 172.21.15.63:0/1947317375 conn(0x7f57680f7e50 msgr2=0x7f57680fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- 172.21.15.63:0/1947317375 shutdown_connections 2024-06-15T00:06:46.072 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- 172.21.15.63:0/1947317375 wait complete. 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 Processor -- start 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.063+0000 7f576e67c640 1 -- start start 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f576806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5768070280 con 0x7f5768102c40 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57680703f0 con 0x7f57681022a0 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:55582/0 (socket says 172.21.15.63:55582) 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 -- 172.21.15.63:0/614277769 learned_addr learned my addr 172.21.15.63:0/614277769 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f5767fff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f576806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 -- 172.21.15.63:0/614277769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 msgr2=0x7f576806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f576806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.073 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 -- 172.21.15.63:0/614277769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5750005040 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f5767fff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f576806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f575ffff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5750000c00 tx=0x7f5750009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5750038070 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57500096f0 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57681192a0 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5750032da0 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.064+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5768119620 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.065+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57680fda10 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.066+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f5750030020 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.066+0000 7f57657fa640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 0x7f57540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.066+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f57500a2ab0 con 0x7f5768102c40 2024-06-15T00:06:46.074 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.066+0000 7f5767fff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 0x7f57540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:46.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.067+0000 7f5767fff640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 0x7f57540678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5758009a80 tx=0x7f5758008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:46.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.070+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5750070f60 con 0x7f5768102c40 2024-06-15T00:06:46.239 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.241+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f576804b480 con 0x7f5754065430 2024-06-15T00:06:46.240 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.242+0000 7f57657fa640 1 -- 172.21.15.63:0/614277769 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27970 (secure 0 0 0) 0x7f576804b480 con 0x7f5754065430 2024-06-15T00:06:46.240 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:46.241 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.244+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 msgr2=0x7f57540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.244+0000 7f576e67c640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 0x7f57540678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5758009a80 tx=0x7f5758008040 comp rx=0 tx=0).stop 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.244+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 msgr2=0x7f576806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.244+0000 7f576e67c640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5750000c00 tx=0x7f5750009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.244+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 shutdown_connections 2024-06-15T00:06:46.242 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5754065430 0x7f57540678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f5768102c40 0x7f576806fcb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 --2- 172.21.15.63:0/614277769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57681022a0 0x7f576806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:46.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 >> 172.21.15.63:0/614277769 conn(0x7f57680f7e50 msgr2=0x7f57680f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:46.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 shutdown_connections 2024-06-15T00:06:46.243 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:46.245+0000 7f576e67c640 1 -- 172.21.15.63:0/614277769 wait complete. 2024-06-15T00:06:46.919 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-06-15T00:06:45.894054+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788020,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682892,"statfs":{"total":575970213888,"available":574139281408,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002176"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-15T00:05:57.047661+0000","last_change":"2024-06-15T00:05:31.205859+0000","last_active":"2024-06-15T00:05:57.047661+0000","last_peered":"2024-06-15T00:05:57.047661+0000","last_clean":"2024-06-15T00:05:57.047661+0000","last_became_active":"2024-06-15T00:05:31.205499+0000","last_became_peered":"2024-06-15T00:05:31.205499+0000","last_unstale":"2024-06-15T00:05:57.047661+0000","last_undegraded":"2024-06-15T00:05:57.047661+0000","last_fullsized":"2024-06-15T00:05:57.047661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T00:04:44.337840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T00:04:44.337840+0000","last_clean_scrub_stamp":"2024-06-15T00:04:44.337840+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:45:34.597441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888080,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.69599999999999995}]},{"osd":2,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.63100000000000001}]},{"osd":3,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.502}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.58399999999999996}]},{"osd":1,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.63600000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.62},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.56200000000000006}]},{"osd":4,"last update":"Sat Jun 15 00:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.41599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36599999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.31}]},{"osd":1,"last update":"Sat Jun 15 00:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Jun 15 00:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.34999999999999998,"15min":0.34300000000000003},"min":{"1min":0.26400000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61399999999999999}]},{"osd":4,"last update":"Sat Jun 15 00:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.629},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 15 00:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.318,"15min":0.314},"min":{"1min":0.23999999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.44800000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33300000000000002},"min":{"1min":0.252,"5min":0.219,"15min":0.219},"max":{"1min":0.61599999999999999,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.432}]},{"osd":4,"last update":"Sat Jun 15 00:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.443}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T00:06:46.920 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-15T00:06:47.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:46 smithi063 ceph-mon[28882]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:47.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:47.186 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:47.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:46 smithi136 ceph-mon[35883]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:47.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:06:48.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:48 smithi136 ceph-mon[35883]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:06:48.423 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:48 smithi063 ceph-mon[28882]: from='client.14416 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- 172.21.15.63:0/3770916130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4738102c60 msgr2=0x7f47380fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 --2- 172.21.15.63:0/3770916130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4738102c60 0x7f47380fc2f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f472c0048d0 tx=0x7f472c009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- 172.21.15.63:0/3770916130 shutdown_connections 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 --2- 172.21.15.63:0/3770916130 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4738102c60 0x7f47380fc2f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 --2- 172.21.15.63:0/3770916130 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47381022c0 0x7f4738102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- 172.21.15.63:0/3770916130 >> 172.21.15.63:0/3770916130 conn(0x7f47380f7e50 msgr2=0x7f47380fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- 172.21.15.63:0/3770916130 shutdown_connections 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- 172.21.15.63:0/3770916130 wait complete. 2024-06-15T00:06:48.910 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 Processor -- start 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.910+0000 7f473d66c640 1 -- start start 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f473d66c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f473d66c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4738102c60 0x7f4738113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f473d66c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4738113ec0 con 0x7f47381022c0 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f473d66c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f473810d270 con 0x7f4738102c60 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48010/0 (socket says 172.21.15.63:48010) 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 -- 172.21.15.63:0/3931936206 learned_addr learned my addr 172.21.15.63:0/3931936206 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f47367fc640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4738102c60 0x7f4738113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 -- 172.21.15.63:0/3931936206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4738102c60 msgr2=0x7f4738113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4738102c60 0x7f4738113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:48.911 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 -- 172.21.15.63:0/3931936206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c005040 con 0x7f47381022c0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.911+0000 7f4736ffd640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f472800b790 tx=0x7f47280091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.912+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f472800c700 con 0x7f47381022c0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.912+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f473810d550 con 0x7f47381022c0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.912+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f472800cd10 con 0x7f47381022c0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.912+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4728013370 con 0x7f47381022c0 2024-06-15T00:06:48.912 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.912+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f473810da70 con 0x7f47381022c0 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.913+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47380fda30 con 0x7f47381022c0 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.913+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4728012070 con 0x7f47381022c0 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.913+0000 7f471f7fe640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 0x7f47200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.914+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4728081e70 con 0x7f47381022c0 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.914+0000 7f47367fc640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 0x7f47200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:48.913 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.914+0000 7f47367fc640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 0x7f47200678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f472c007ac0 tx=0x7f472c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:48.915 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:48.917+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4728050320 con 0x7f47381022c0 2024-06-15T00:06:49.087 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.090+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 --> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f473804b4b0 con 0x7f4720065430 2024-06-15T00:06:49.088 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.091+0000 7f471f7fe640 1 -- 172.21.15.63:0/3931936206 <== mgr.14212 v2:172.21.15.63:6800/3568086764 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27971 (secure 0 0 0) 0x7f473804b4b0 con 0x7f4720065430 2024-06-15T00:06:49.089 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:49.089 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2024-06-15T00:06:49.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 msgr2=0x7f47200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:49.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 0x7f47200678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f472c007ac0 tx=0x7f472c035040 comp rx=0 tx=0).stop 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 msgr2=0x7f47381132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f472800b790 tx=0x7f47280091d0 comp rx=0 tx=0).stop 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 shutdown_connections 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4720065430 0x7f47200678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4738102c60 0x7f4738113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:49.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 --2- 172.21.15.63:0/3931936206 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47381022c0 0x7f47381132f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:49.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 >> 172.21.15.63:0/3931936206 conn(0x7f47380f7e50 msgr2=0x7f47380f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:49.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 shutdown_connections 2024-06-15T00:06:49.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:49.093+0000 7f473d66c640 1 -- 172.21.15.63:0/3931936206 wait complete. 2024-06-15T00:06:49.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:49 smithi136 ceph-mon[35883]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:49.388 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:49 smithi063 ceph-mon[28882]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:49.716 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-06-15T00:06:47.894402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788020,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682892,"statfs":{"total":575970213888,"available":574139281408,"internally_reserved":0,"allocated":3969024,"data_stored":2606262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002178"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":190,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-15T00:05:57.047661+0000","last_change":"2024-06-15T00:05:31.205859+0000","last_active":"2024-06-15T00:05:57.047661+0000","last_peered":"2024-06-15T00:05:57.047661+0000","last_clean":"2024-06-15T00:05:57.047661+0000","last_became_active":"2024-06-15T00:05:31.205499+0000","last_became_peered":"2024-06-15T00:05:31.205499+0000","last_unstale":"2024-06-15T00:05:57.047661+0000","last_undegraded":"2024-06-15T00:05:57.047661+0000","last_fullsized":"2024-06-15T00:05:57.047661+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T00:04:44.337840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T00:04:44.337840+0000","last_clean_scrub_stamp":"2024-06-15T00:04:44.337840+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T08:45:34.597441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50900000000000001}]}]},{"osd":4,"up_from":34,"seq":146028888081,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"last":0.438}]},{"osd":1,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52600000000000002}]},{"osd":2,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.46600000000000003}]},{"osd":3,"last update":"Sat Jun 15 00:06:31 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.40699999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149720,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.54800000000000004}]},{"osd":1,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.502}]},{"osd":2,"last update":"Sat Jun 15 00:06:10 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.45200000000000001}]},{"osd":4,"last update":"Sat Jun 15 00:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.37,"15min":0.37},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.34100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30499999999999999}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":139193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.5,"5min":0.5,"15min":0.5},"last":0.31}]},{"osd":1,"last update":"Sat Jun 15 00:05:44 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.47599999999999998}]},{"osd":4,"last update":"Sat Jun 15 00:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41299999999999998}]}]},{"osd":1,"up_from":14,"seq":60129542176,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:06:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.33700000000000002,"15min":0.33300000000000002},"min":{"1min":0.217,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.34999999999999998,"15min":0.34300000000000003},"min":{"1min":0.26400000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.109,"5min":1.109,"15min":1.109},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"last":0.442},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.47199999999999998}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.61399999999999999}]},{"osd":4,"last update":"Sat Jun 15 00:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.45,"5min":1.45,"15min":1.45},"last":0.629},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.52000000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 15 00:06:19 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.318,"15min":0.314},"min":{"1min":0.23999999999999999,"5min":0.187,"15min":0.187},"max":{"1min":0.44800000000000001,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.37,"5min":0.33800000000000002,"15min":0.33300000000000002},"min":{"1min":0.252,"5min":0.219,"15min":0.219},"max":{"1min":0.61599999999999999,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.23400000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:05:45 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.33200000000000002}]},{"osd":3,"last update":"Sat Jun 15 00:06:07 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.432}]},{"osd":4,"last update":"Sat Jun 15 00:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.5980000000000001,"5min":1.5980000000000001,"15min":1.5980000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.8420000000000001,"5min":1.8420000000000001,"15min":1.8420000000000001},"last":0.443}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T00:06:49.717 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-15T00:06:49.717 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-15T00:06:49.718 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-15T00:06:49.718 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:06:49.987 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:50.332 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:50 smithi063 ceph-mon[28882]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:06:50.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:50 smithi136 ceph-mon[35883]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:06:51.242 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:51 smithi063 ceph-mon[28882]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:51.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:51 smithi136 ceph-mon[35883]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:51.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 -- 172.21.15.63:0/3470431574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 msgr2=0x7fd1e80fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/3470431574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e80fc300 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80048d0 tx=0x7fd1d8009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 -- 172.21.15.63:0/3470431574 shutdown_connections 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/3470431574 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e80fc300 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/3470431574 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 0x7fd1e81026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 -- 172.21.15.63:0/3470431574 >> 172.21.15.63:0/3470431574 conn(0x7fd1e80f7e80 msgr2=0x7fd1e80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 -- 172.21.15.63:0/3470431574 shutdown_connections 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.467+0000 7fd1ecbac640 1 -- 172.21.15.63:0/3470431574 wait complete. 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 Processor -- start 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 -- start start 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 0x7fd1e8112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:51.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e8113aa0 con 0x7fd1e8102c70 2024-06-15T00:06:51.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1ecbac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1e810cee0 con 0x7fd1e81022d0 2024-06-15T00:06:51.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1e5d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:51.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1e5d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48034/0 (socket says 172.21.15.63:48034) 2024-06-15T00:06:51.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/1252506360 learned_addr learned my addr 172.21.15.63:0/1252506360 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:51.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1e659c640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 0x7fd1e8112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:51.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.468+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/1252506360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 msgr2=0x7fd1e8112f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:51.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1e5d9b640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 0x7fd1e8112f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1e5d9b640 1 -- 172.21.15.63:0/1252506360 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8005040 con 0x7fd1e8102c70 2024-06-15T00:06:51.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1e5d9b640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80048a0 tx=0x7fd1d8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:51.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d8038070 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd1d802cef0 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e810d160 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd1d8009640 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.469+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e810d6a0 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.470+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e80fda40 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.470+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd1d8030020 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.471+0000 7fd1deffd640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 0x7fd1d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.471+0000 7fd1e659c640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 0x7fd1d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.471+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd1d80a27f0 con 0x7fd1e8102c70 2024-06-15T00:06:51.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.471+0000 7fd1e659c640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 0x7fd1d40678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0004ed0 tx=0x7fd1d000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:51.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.475+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd1d8070d20 con 0x7fd1e8102c70 2024-06-15T00:06:51.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.682+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd1e80fbac0 con 0x7fd1e8102c70 2024-06-15T00:06:51.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.682+0000 7fd1deffd640 1 -- 172.21.15.63:0/1252506360 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd1d8074340 con 0x7fd1e8102c70 2024-06-15T00:06:51.680 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:51.680 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:06:51.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 msgr2=0x7fd1d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:51.681 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 0x7fd1d40678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd1d0004ed0 tx=0x7fd1d000b040 comp rx=0 tx=0).stop 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 msgr2=0x7fd1e81134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd1d80048a0 tx=0x7fd1d8009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 shutdown_connections 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd1d4065430 0x7fd1d40678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd1e8102c70 0x7fd1e81134a0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 --2- 172.21.15.63:0/1252506360 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1e81022d0 0x7fd1e8112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 >> 172.21.15.63:0/1252506360 conn(0x7fd1e80f7e80 msgr2=0x7fd1e80f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 shutdown_connections 2024-06-15T00:06:51.682 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:51.684+0000 7fd1ecbac640 1 -- 172.21.15.63:0/1252506360 wait complete. 2024-06-15T00:06:52.325 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:52 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1252506360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:06:52.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:52 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1252506360' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:06:53.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:53 smithi136 ceph-mon[35883]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:53.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:53 smithi063 ceph-mon[28882]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:55.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:55 smithi136 ceph-mon[35883]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:55.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:06:55.382 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:55 smithi063 ceph-mon[28882]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:55.630 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:06:57.123 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:57 smithi063 ceph-mon[28882]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:57.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:57 smithi136 ceph-mon[35883]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:57.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 -- 172.21.15.63:0/2494061274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 msgr2=0x7f03680fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:57.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 --2- 172.21.15.63:0/2494061274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03680fb1e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f03540048d0 tx=0x7f0354009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 -- 172.21.15.63:0/2494061274 shutdown_connections 2024-06-15T00:06:57.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 --2- 172.21.15.63:0/2494061274 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03680fb1e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.358 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 --2- 172.21.15.63:0/2494061274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 0x7f0368101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 -- 172.21.15.63:0/2494061274 >> 172.21.15.63:0/2494061274 conn(0x7f03680f6d80 msgr2=0x7f03680f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 -- 172.21.15.63:0/2494061274 shutdown_connections 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.359+0000 7f036f176640 1 -- 172.21.15.63:0/2494061274 wait complete. 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 Processor -- start 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 -- start start 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 0x7f0368106070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368106b80 con 0x7f0368101b50 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036f176640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0368106cf0 con 0x7f03681011b0 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f0367fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f0367fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48056/0 (socket says 172.21.15.63:48056) 2024-06-15T00:06:57.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f0367fff640 1 -- 172.21.15.63:0/532810195 learned_addr learned my addr 172.21.15.63:0/532810195 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:06:57.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.360+0000 7f036cf12640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 0x7f0368106070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:57.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f0367fff640 1 -- 172.21.15.63:0/532810195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 msgr2=0x7f0368106070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:57.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f0367fff640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 0x7f0368106070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f0367fff640 1 -- 172.21.15.63:0/532810195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0354005040 con 0x7f0368101b50 2024-06-15T00:06:57.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f0367fff640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f03540048a0 tx=0x7f0354009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0354038070 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03681192f0 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0354032d30 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0354009680 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.361+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0368119670 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.362+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03680fc920 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.363+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0354030020 con 0x7f0368101b50 2024-06-15T00:06:57.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.363+0000 7f03657fa640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 0x7f03580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:06:57.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.363+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f03540a2300 con 0x7f0368101b50 2024-06-15T00:06:57.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.363+0000 7f036cf12640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 0x7f03580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:06:57.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.364+0000 7f036cf12640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 0x7f03580678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f035c006fd0 tx=0x7f035c008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:06:57.364 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.367+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f03540707b0 con 0x7f0368101b50 2024-06-15T00:06:57.572 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.575+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f03680fa9a0 con 0x7f0368101b50 2024-06-15T00:06:57.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.575+0000 7f03657fa640 1 -- 172.21.15.63:0/532810195 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0354073dd0 con 0x7f0368101b50 2024-06-15T00:06:57.573 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:06:57.573 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 msgr2=0x7f03580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 0x7f03580678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f035c006fd0 tx=0x7f035c008040 comp rx=0 tx=0).stop 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 msgr2=0x7f03681065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f03540048a0 tx=0x7f0354009da0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 shutdown_connections 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0358065430 0x7f03580678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0368101b50 0x7f03681065b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 --2- 172.21.15.63:0/532810195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03681011b0 0x7f0368106070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 >> 172.21.15.63:0/532810195 conn(0x7f03680f6d80 msgr2=0x7f03680f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:06:57.575 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 shutdown_connections 2024-06-15T00:06:57.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:06:57.577+0000 7f036f176640 1 -- 172.21.15.63:0/532810195 wait complete. 2024-06-15T00:06:58.146 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:58 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/532810195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:06:58.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:58 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/532810195' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:06:59.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:06:59 smithi136 ceph-mon[35883]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:06:59.426 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:06:59 smithi063 ceph-mon[28882]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:00.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:07:00.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:07:00.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:07:00.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:07:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:07:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:07:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:07:00.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:07:01.177 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:01 smithi136 ceph-mon[35883]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:01 smithi063 ceph-mon[28882]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:01.457 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:02.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:02.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:03.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1728041461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 msgr2=0x7fd7e00fae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:03.181 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1728041461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e00fae50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fd7d00048d0 tx=0x7fd7d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1728041461 shutdown_connections 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1728041461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e00fae50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1728041461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 0x7fd7e00fd540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1728041461 >> 172.21.15.63:0/1728041461 conn(0x7fd7e0070a40 msgr2=0x7fd7e0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1728041461 shutdown_connections 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.172+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1728041461 wait complete. 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 Processor -- start 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 -- start start 2024-06-15T00:07:03.182 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 0x7fd7e0115630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e0115c30 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7e5f20640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7e010f070 con 0x7fd7e00fd170 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7df7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:44656/0 (socket says 172.21.15.63:44656) 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.173+0000 7fd7df7fe640 1 -- 172.21.15.63:0/1226645057 learned_addr learned my addr 172.21.15.63:0/1226645057 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7deffd640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 0x7fd7e0115630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7df7fe640 1 -- 172.21.15.63:0/1226645057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 msgr2=0x7fd7e0115630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7df7fe640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 0x7fd7e0115630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.183 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7df7fe640 1 -- 172.21.15.63:0/1226645057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d0005040 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7df7fe640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8002a50 tx=0x7fd7c8004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c8007c30 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7e010f350 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7c800f040 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7c800da70 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.174+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7e010f8f0 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.175+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e00fb9f0 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.176+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fd7c800dc10 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.176+0000 7fd7d7fff640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 0x7fd7cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.176+0000 7fd7deffd640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 0x7fd7cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.176+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd7c8081d70 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.177+0000 7fd7deffd640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 0x7fd7cc0678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7e01109b0 tx=0x7fd7d0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:03.184 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.181+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd7c804cdf0 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:03 smithi136 ceph-mon[35883]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:03.386 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.389+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd7e004b480 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:03 smithi063 ceph-mon[28882]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:03.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.389+0000 7fd7d7fff640 1 -- 172.21.15.63:0/1226645057 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd7c8050220 con 0x7fd7e00fa9e0 2024-06-15T00:07:03.387 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:03.387 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 msgr2=0x7fd7cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 0x7fd7cc0678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd7e01109b0 tx=0x7fd7d0035040 comp rx=0 tx=0).stop 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 msgr2=0x7fd7e01150f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8002a50 tx=0x7fd7c8004600 comp rx=0 tx=0).stop 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 shutdown_connections 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fd7cc065430 0x7fd7cc0678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7e00fd170 0x7fd7e0115630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 --2- 172.21.15.63:0/1226645057 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fd7e00fa9e0 0x7fd7e01150f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 >> 172.21.15.63:0/1226645057 conn(0x7fd7e0070a40 msgr2=0x7fd7e00f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:03.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 shutdown_connections 2024-06-15T00:07:03.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:03.391+0000 7fd7e5f20640 1 -- 172.21.15.63:0/1226645057 wait complete. 2024-06-15T00:07:04.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:04 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1226645057' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:04.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:04 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1226645057' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:05.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:05 smithi136 ceph-mon[35883]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:05.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:05 smithi063 ceph-mon[28882]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:07.064 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:07.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:07 smithi136 ceph-mon[35883]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:07.343 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:07.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:07 smithi063 ceph-mon[28882]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:08.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.899+0000 7fae442db640 1 -- 172.21.15.63:0/276371083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fb570 msgr2=0x7fae3c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:08.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 --2- 172.21.15.63:0/276371083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fb570 0x7fae3c0fb940 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009c50 tx=0x7fae2c004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:08.898 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- 172.21.15.63:0/276371083 shutdown_connections 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 --2- 172.21.15.63:0/276371083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 --2- 172.21.15.63:0/276371083 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fb570 0x7fae3c0fb940 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- 172.21.15.63:0/276371083 >> 172.21.15.63:0/276371083 conn(0x7fae3c072110 msgr2=0x7fae3c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- 172.21.15.63:0/276371083 shutdown_connections 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- 172.21.15.63:0/276371083 wait complete. 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 Processor -- start 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- start start 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fb570 0x7fae3c0f8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.900+0000 7fae442db640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0f9230 con 0x7fae3c0fbe80 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae442db640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0f93a0 con 0x7fae3c0fb570 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35324/0 (socket says 172.21.15.63:35324) 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 -- 172.21.15.63:0/1464526596 learned_addr learned my addr 172.21.15.63:0/1464526596 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:08.899 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae42077640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fb570 0x7fae3c0f8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:08.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 -- 172.21.15.63:0/1464526596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fb570 msgr2=0x7fae3c0f8720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:08.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fb570 0x7fae3c0f8720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:08.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 -- 172.21.15.63:0/1464526596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c0098b0 con 0x7fae3c0fbe80 2024-06-15T00:07:08.900 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae41876640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae3000b790 tx=0x7fae300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae3000c700 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae3000cd10 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c1192c0 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fae30013370 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.901+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c119700 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.902+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c0fc920 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.903+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fae30012070 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.903+0000 7fae3affd640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 0x7fae240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.903+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fae30082140 con 0x7fae3c0fbe80 2024-06-15T00:07:08.901 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.903+0000 7fae42077640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 0x7fae240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:08.902 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.904+0000 7fae42077640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 0x7fae240678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009c20 tx=0x7fae2c004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:08.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:08.908+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fae300505f0 con 0x7fae3c0fbe80 2024-06-15T00:07:09.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.117+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fae3c0f9e10 con 0x7fae3c0fbe80 2024-06-15T00:07:09.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.118+0000 7fae3affd640 1 -- 172.21.15.63:0/1464526596 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fae30053c10 con 0x7fae3c0fbe80 2024-06-15T00:07:09.115 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:09.115 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 msgr2=0x7fae240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 0x7fae240678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fae2c009c20 tx=0x7fae2c004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 msgr2=0x7fae3c0f8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fae3000b790 tx=0x7fae300091d0 comp rx=0 tx=0).stop 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 shutdown_connections 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fae24065430 0x7fae240678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fae3c0fbe80 0x7fae3c0f8c60 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 --2- 172.21.15.63:0/1464526596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae3c0fb570 0x7fae3c0f8720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:09.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.119+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 >> 172.21.15.63:0/1464526596 conn(0x7fae3c072110 msgr2=0x7fae3c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:09.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.120+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 shutdown_connections 2024-06-15T00:07:09.118 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:09.120+0000 7fae442db640 1 -- 172.21.15.63:0/1464526596 wait complete. 2024-06-15T00:07:09.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:09 smithi136 ceph-mon[35883]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:09.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:09 smithi063 ceph-mon[28882]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:10.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:10 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1464526596' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:10.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:10 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1464526596' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:11.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:11 smithi136 ceph-mon[35883]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:11.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:11 smithi063 ceph-mon[28882]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:12.725 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:13.002 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:13 smithi136 ceph-mon[35883]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:13.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:13 smithi063 ceph-mon[28882]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:14.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 -- 172.21.15.63:0/867197655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 msgr2=0x7fe1c00fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:14.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/867197655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c00fc2d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0048d0 tx=0x7fe1bc009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 -- 172.21.15.63:0/867197655 shutdown_connections 2024-06-15T00:07:14.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/867197655 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c00fc2d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.698 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/867197655 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 -- 172.21.15.63:0/867197655 >> 172.21.15.63:0/867197655 conn(0x7fe1c00f7e50 msgr2=0x7fe1c00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 -- 172.21.15.63:0/867197655 shutdown_connections 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.680+0000 7fe1c7c53640 1 -- 172.21.15.63:0/867197655 wait complete. 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 Processor -- start 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 -- start start 2024-06-15T00:07:14.699 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0112f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c0113aa0 con 0x7fe1c0102c40 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c7c53640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1c010cee0 con 0x7fe1c01022a0 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c51ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c51ee640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35354/0 (socket says 172.21.15.63:35354) 2024-06-15T00:07:14.700 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c51ee640 1 -- 172.21.15.63:0/2187614397 learned_addr learned my addr 172.21.15.63:0/2187614397 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c59ef640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0112f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c51ee640 1 -- 172.21.15.63:0/2187614397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 msgr2=0x7fe1c0112f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.681+0000 7fe1c51ee640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0112f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1c51ee640 1 -- 172.21.15.63:0/2187614397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1bc005040 con 0x7fe1c0102c40 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1c59ef640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1c51ee640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0048a0 tx=0x7fe1bc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:14.701 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1bc038070 con 0x7fe1c0102c40 2024-06-15T00:07:14.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1c010d160 con 0x7fe1c0102c40 2024-06-15T00:07:14.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1bc02cef0 con 0x7fe1c0102c40 2024-06-15T00:07:14.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1bc009640 con 0x7fe1c0102c40 2024-06-15T00:07:14.702 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.682+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1c010d6a0 con 0x7fe1c0102c40 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.683+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c00fda10 con 0x7fe1c0102c40 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.683+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe1bc030020 con 0x7fe1c0102c40 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.684+0000 7fe1b27fc640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 0x7fe1ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.684+0000 7fe1c59ef640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 0x7fe1ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.684+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe1bc0a27d0 con 0x7fe1c0102c40 2024-06-15T00:07:14.703 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.684+0000 7fe1c59ef640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 0x7fe1ac0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4007fd0 tx=0x7fe1b4009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:14.704 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.688+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe1bc070c80 con 0x7fe1c0102c40 2024-06-15T00:07:14.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.896+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe1c01092c0 con 0x7fe1c0102c40 2024-06-15T00:07:14.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.896+0000 7fe1b27fc640 1 -- 172.21.15.63:0/2187614397 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe1bc009870 con 0x7fe1c0102c40 2024-06-15T00:07:14.894 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:14.894 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:14.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 msgr2=0x7fe1ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:14.896 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 0x7fe1ac0678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4007fd0 tx=0x7fe1b4009e90 comp rx=0 tx=0).stop 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 msgr2=0x7fe1c01134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc0048a0 tx=0x7fe1bc009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 shutdown_connections 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1ac065430 0x7fe1ac0678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1c0102c40 0x7fe1c01134a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 --2- 172.21.15.63:0/2187614397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1c01022a0 0x7fe1c0112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 >> 172.21.15.63:0/2187614397 conn(0x7fe1c00f7e50 msgr2=0x7fe1c00f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 shutdown_connections 2024-06-15T00:07:14.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:14.899+0000 7fe1c7c53640 1 -- 172.21.15.63:0/2187614397 wait complete. 2024-06-15T00:07:15.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:15 smithi063 ceph-mon[28882]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:15.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:15 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2187614397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:15.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:15 smithi136 ceph-mon[35883]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:15.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:15 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2187614397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:17.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:17 smithi136 ceph-mon[35883]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:17.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:17 smithi063 ceph-mon[28882]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:18.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:18.768 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:19.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:19 smithi063 ceph-mon[28882]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:19.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:19 smithi136 ceph-mon[35883]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:20.259 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 -- 172.21.15.63:0/1370301904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fbe80 msgr2=0x7f93f00fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 --2- 172.21.15.63:0/1370301904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fbe80 0x7f93f00fcd80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009c50 tx=0x7f93e4004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 -- 172.21.15.63:0/1370301904 shutdown_connections 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 --2- 172.21.15.63:0/1370301904 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fbe80 0x7f93f00fcd80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 --2- 172.21.15.63:0/1370301904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fb570 0x7f93f00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 -- 172.21.15.63:0/1370301904 >> 172.21.15.63:0/1370301904 conn(0x7f93f0072110 msgr2=0x7f93f0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 -- 172.21.15.63:0/1370301904 shutdown_connections 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 -- 172.21.15.63:0/1370301904 wait complete. 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.251+0000 7f93f5c79640 1 Processor -- start 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93f5c79640 1 -- start start 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93f5c79640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:20.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93f5c79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fbe80 0x7f93f0115650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93f5c79640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f0115c50 con 0x7f93f00fb570 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93f5c79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f010f0e0 con 0x7f93f00fbe80 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47980/0 (socket says 172.21.15.63:47980) 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 -- 172.21.15.63:0/2978882462 learned_addr learned my addr 172.21.15.63:0/2978882462 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93eeffd640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fbe80 0x7f93f0115650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 -- 172.21.15.63:0/2978882462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fbe80 msgr2=0x7f93f0115650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fbe80 0x7f93f0115650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 -- 172.21.15.63:0/2978882462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e40098b0 con 0x7f93f00fb570 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.252+0000 7f93ef7fe640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c790 tx=0x7f93e00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.253+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e000a820 con 0x7f93f00fb570 2024-06-15T00:07:20.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.253+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93e000ae30 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.253+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f010f3c0 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.253+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f93e00113a0 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.253+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f010f930 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.254+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f00fc920 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.254+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f93e0012400 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.255+0000 7f93d7fff640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 0x7f93d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.255+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f93e0082580 con 0x7f93f00fb570 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.255+0000 7f93eeffd640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 0x7f93d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.255+0000 7f93eeffd640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 0x7f93d80678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009c20 tx=0x7f93e4004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:20.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.259+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f93e0050c10 con 0x7f93f00fb570 2024-06-15T00:07:20.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.469+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f93f0072c20 con 0x7f93f00fb570 2024-06-15T00:07:20.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.469+0000 7f93d7fff640 1 -- 172.21.15.63:0/2978882462 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f93e0054230 con 0x7f93f00fb570 2024-06-15T00:07:20.467 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:20.467 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:20.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 msgr2=0x7f93d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:20.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 0x7f93d80678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f93e4009c20 tx=0x7f93e4004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:20.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 msgr2=0x7f93f0115110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:20.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f93e000c790 tx=0x7f93e00043c0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 shutdown_connections 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f93d8065430 0x7f93d80678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93f00fbe80 0x7f93f0115650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 --2- 172.21.15.63:0/2978882462 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f93f00fb570 0x7f93f0115110 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.471+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 >> 172.21.15.63:0/2978882462 conn(0x7f93f0072110 msgr2=0x7f93f00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.472+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 shutdown_connections 2024-06-15T00:07:20.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:20.472+0000 7f93f5c79640 1 -- 172.21.15.63:0/2978882462 wait complete. 2024-06-15T00:07:21.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:21 smithi063 ceph-mon[28882]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:21.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:21 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2978882462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:21.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:21 smithi136 ceph-mon[35883]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:21.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:21 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2978882462' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:23.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:23 smithi136 ceph-mon[35883]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:23.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:23 smithi063 ceph-mon[28882]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:24.065 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:24.340 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:25.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:25 smithi063 ceph-mon[28882]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:25.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:25 smithi136 ceph-mon[35883]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:25.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 -- 172.21.15.63:0/3095339410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 msgr2=0x7f33f00fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 --2- 172.21.15.63:0/3095339410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f00fb200 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f33e00048d0 tx=0x7f33e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 -- 172.21.15.63:0/3095339410 shutdown_connections 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 --2- 172.21.15.63:0/3095339410 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f00fb200 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 --2- 172.21.15.63:0/3095339410 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f01015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 -- 172.21.15.63:0/3095339410 >> 172.21.15.63:0/3095339410 conn(0x7f33f00f6dc0 msgr2=0x7f33f00f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 -- 172.21.15.63:0/3095339410 shutdown_connections 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.930+0000 7f33f7b36640 1 -- 172.21.15.63:0/3095339410 wait complete. 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 Processor -- start 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 -- start start 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f0115170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f0115cb0 con 0x7f33f0101b70 2024-06-15T00:07:25.929 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f7b36640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f010f0f0 con 0x7f33f01011d0 2024-06-15T00:07:25.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f50d1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:25.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f50d1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47998/0 (socket says 172.21.15.63:47998) 2024-06-15T00:07:25.930 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.931+0000 7f33f50d1640 1 -- 172.21.15.63:0/2520314236 learned_addr learned my addr 172.21.15.63:0/2520314236 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f58d2640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f0115170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f50d1640 1 -- 172.21.15.63:0/2520314236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 msgr2=0x7f33f0115170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f50d1640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f0115170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f50d1640 1 -- 172.21.15.63:0/2520314236 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e0005040 con 0x7f33f0101b70 2024-06-15T00:07:25.931 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f58d2640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f0115170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f50d1640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33e00048a0 tx=0x7f33e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e0038070 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f33e0032d30 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f010f370 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e0009680 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.932+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f010f830 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.933+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f00fc940 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.934+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f33e0030020 con 0x7f33f0101b70 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.934+0000 7f33e67fc640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 0x7f33dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.934+0000 7f33f58d2640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 0x7f33dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:25.932 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.934+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f33e00a21f0 con 0x7f33f0101b70 2024-06-15T00:07:25.933 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.935+0000 7f33f58d2640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 0x7f33dc0678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33ec004c80 tx=0x7f33ec009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:25.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:25.938+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f33e0070720 con 0x7f33f0101b70 2024-06-15T00:07:26.149 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.151+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f33f0110840 con 0x7f33f0101b70 2024-06-15T00:07:26.150 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.152+0000 7f33e67fc640 1 -- 172.21.15.63:0/2520314236 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f33e0073d40 con 0x7f33f0101b70 2024-06-15T00:07:26.153 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:26.153 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:26.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 msgr2=0x7f33dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:26.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 0x7f33dc0678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f33ec004c80 tx=0x7f33ec009040 comp rx=0 tx=0).stop 2024-06-15T00:07:26.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 msgr2=0x7f33f01156b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:26.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f33e00048a0 tx=0x7f33e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:26.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 shutdown_connections 2024-06-15T00:07:26.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f33dc065430 0x7f33dc0678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:26.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f33f0101b70 0x7f33f01156b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:26.156 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 --2- 172.21.15.63:0/2520314236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f01011d0 0x7f33f0115170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:26.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 >> 172.21.15.63:0/2520314236 conn(0x7f33f00f6dc0 msgr2=0x7f33f00f84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:26.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 shutdown_connections 2024-06-15T00:07:26.157 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:26.154+0000 7f33f7b36640 1 -- 172.21.15.63:0/2520314236 wait complete. 2024-06-15T00:07:27.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:27 smithi063 ceph-mon[28882]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:27.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:27 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2520314236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:27.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:27 smithi136 ceph-mon[35883]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:27.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:27 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2520314236' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:29.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:29 smithi063 ceph-mon[28882]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:29.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:29 smithi136 ceph-mon[35883]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:29.728 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:30.009 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:31.231 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:31 smithi063 ceph-mon[28882]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:31.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:31 smithi136 ceph-mon[35883]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 -- 172.21.15.63:0/1097934936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101200 msgr2=0x7f47a01015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 --2- 172.21.15.63:0/1097934936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101200 0x7f47a01015d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f47900048d0 tx=0x7f4790009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 -- 172.21.15.63:0/1097934936 shutdown_connections 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 --2- 172.21.15.63:0/1097934936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101ba0 0x7f47a00fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 --2- 172.21.15.63:0/1097934936 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101200 0x7f47a01015d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 -- 172.21.15.63:0/1097934936 >> 172.21.15.63:0/1097934936 conn(0x7f47a00f6df0 msgr2=0x7f47a00f9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 -- 172.21.15.63:0/1097934936 shutdown_connections 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 -- 172.21.15.63:0/1097934936 wait complete. 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.862+0000 7f47a4e93640 1 Processor -- start 2024-06-15T00:07:31.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f47a4e93640 1 -- start start 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f47a4e93640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101200 0x7f47a0112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f47a4e93640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f47a4e93640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0113b50 con 0x7f47a0101ba0 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f47a4e93640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a010cf40 con 0x7f47a0101200 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54386/0 (socket says 172.21.15.63:54386) 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 -- 172.21.15.63:0/569115994 learned_addr learned my addr 172.21.15.63:0/569115994 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479e59c640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101200 0x7f47a0112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:31.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 -- 172.21.15.63:0/569115994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101200 msgr2=0x7f47a0112f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101200 0x7f47a0112f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.863+0000 7f479dd9b640 1 -- 172.21.15.63:0/569115994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4790005040 con 0x7f47a0101ba0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f479dd9b640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f479400b790 tx=0x7f47940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f479400c700 con 0x7f47a0101ba0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a010d220 con 0x7f47a0101ba0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f479400cd10 con 0x7f47a0101ba0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4794013370 con 0x7f47a0101ba0 2024-06-15T00:07:31.863 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.864+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a010d740 con 0x7f47a0101ba0 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.865+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00fc970 con 0x7f47a0101ba0 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.865+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4794012070 con 0x7f47a0101ba0 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.866+0000 7f478effd640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 0x7f47880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.866+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4794081ec0 con 0x7f47a0101ba0 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.866+0000 7f479e59c640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 0x7f47880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:31.864 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.866+0000 7f479e59c640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 0x7f47880678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4790004540 tx=0x7f4790035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:31.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:31.871+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4794050370 con 0x7f47a0101ba0 2024-06-15T00:07:32.079 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.082+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47a00fa9f0 con 0x7f47a0101ba0 2024-06-15T00:07:32.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.082+0000 7f478effd640 1 -- 172.21.15.63:0/569115994 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4794053990 con 0x7f47a0101ba0 2024-06-15T00:07:32.080 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:32.080 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 msgr2=0x7f47880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 0x7f47880678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4790004540 tx=0x7f4790035040 comp rx=0 tx=0).stop 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 msgr2=0x7f47a01134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f479400b790 tx=0x7f47940091d0 comp rx=0 tx=0).stop 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 shutdown_connections 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4788065430 0x7f47880678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f47a0101ba0 0x7f47a01134c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 --2- 172.21.15.63:0/569115994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47a0101200 0x7f47a0112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:32.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.084+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 >> 172.21.15.63:0/569115994 conn(0x7f47a00f6df0 msgr2=0x7f47a0065740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:32.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.085+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 shutdown_connections 2024-06-15T00:07:32.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:32.085+0000 7f47a4e93640 1 -- 172.21.15.63:0/569115994 wait complete. 2024-06-15T00:07:32.365 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:32.365 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:32 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/569115994' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:32.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:32.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:32 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/569115994' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:33.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:33 smithi063 ceph-mon[28882]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:33.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:33 smithi136 ceph-mon[35883]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:35.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:35 smithi063 ceph-mon[28882]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:35.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:35 smithi136 ceph-mon[35883]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:35.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:36.154 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:37.401 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:37 smithi063 ceph-mon[28882]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:37.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:37 smithi136 ceph-mon[35883]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 -- 172.21.15.63:0/2231807885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d0102c90 msgr2=0x7f79d00fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 --2- 172.21.15.63:0/2231807885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d0102c90 0x7f79d00fa9e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f79c40048d0 tx=0x7f79c4009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 -- 172.21.15.63:0/2231807885 shutdown_connections 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 --2- 172.21.15.63:0/2231807885 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d0102c90 0x7f79d00fa9e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 --2- 172.21.15.63:0/2231807885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d01022f0 0x7f79d01026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 -- 172.21.15.63:0/2231807885 >> 172.21.15.63:0/2231807885 conn(0x7f79d0070a40 msgr2=0x7f79d0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 -- 172.21.15.63:0/2231807885 shutdown_connections 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.721+0000 7f79d650a640 1 -- 172.21.15.63:0/2231807885 wait complete. 2024-06-15T00:07:37.730 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 Processor -- start 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 -- start start 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d0102c90 0x7f79d0113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d0113b20 con 0x7f79d01022f0 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79d650a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d010cf60 con 0x7f79d0102c90 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79cffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79cffff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54394/0 (socket says 172.21.15.63:54394) 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.722+0000 7f79cffff640 1 -- 172.21.15.63:0/1575401330 learned_addr learned my addr 172.21.15.63:0/1575401330 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79cf7fe640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d0102c90 0x7f79d0113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79cffff640 1 -- 172.21.15.63:0/1575401330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d0102c90 msgr2=0x7f79d0113520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79cffff640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d0102c90 0x7f79d0113520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79cffff640 1 -- 172.21.15.63:0/1575401330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79c4005040 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79cffff640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f79c0002a80 tx=0x7f79c0004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c0007c30 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d010d240 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79c000f040 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79c000daa0 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.723+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d010d730 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.724+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d0065510 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.725+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f79c000dd10 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.725+0000 7f79ccff9640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 0x7f79b80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.725+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f79c0082320 con 0x7f79d01022f0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.725+0000 7f79cf7fe640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 0x7f79b80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.726+0000 7f79cf7fe640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 0x7f79b80678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79d0103190 tx=0x7f79c4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:37.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.729+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f79c00507d0 con 0x7f79d01022f0 2024-06-15T00:07:37.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.938+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f79d0109340 con 0x7f79d01022f0 2024-06-15T00:07:37.935 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.938+0000 7f79ccff9640 1 -- 172.21.15.63:0/1575401330 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f79c0053df0 con 0x7f79d01022f0 2024-06-15T00:07:37.935 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:37.935 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 msgr2=0x7f79b80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 0x7f79b80678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f79d0103190 tx=0x7f79c4035040 comp rx=0 tx=0).stop 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 msgr2=0x7f79d0112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f79c0002a80 tx=0x7f79c0004570 comp rx=0 tx=0).stop 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 shutdown_connections 2024-06-15T00:07:37.937 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f79b8065430 0x7f79b80678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79d0102c90 0x7f79d0113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 --2- 172.21.15.63:0/1575401330 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79d01022f0 0x7f79d0112fe0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:37.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 >> 172.21.15.63:0/1575401330 conn(0x7f79d0070a40 msgr2=0x7f79d00f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:37.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 shutdown_connections 2024-06-15T00:07:37.938 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:37.940+0000 7f79d650a640 1 -- 172.21.15.63:0/1575401330 wait complete. 2024-06-15T00:07:38.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:38 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1575401330' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:38.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:38 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1575401330' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:39.366 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:39 smithi136 ceph-mon[35883]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:39.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:39 smithi063 ceph-mon[28882]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:41.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:41 smithi063 ceph-mon[28882]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:41.578 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:41.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:41 smithi136 ceph-mon[35883]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:41.857 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:43.263 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:43 smithi063 ceph-mon[28882]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:43.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 -- 172.21.15.63:0/792049006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de01011c0 msgr2=0x7f4de0101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:43.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 --2- 172.21.15.63:0/792049006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de01011c0 0x7f4de0101590 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00048d0 tx=0x7f4dd0009da0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.459 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 -- 172.21.15.63:0/792049006 shutdown_connections 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 --2- 172.21.15.63:0/792049006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de0101b60 0x7f4de00fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 --2- 172.21.15.63:0/792049006 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de01011c0 0x7f4de0101590 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 -- 172.21.15.63:0/792049006 >> 172.21.15.63:0/792049006 conn(0x7f4de00f6d70 msgr2=0x7f4de00f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 -- 172.21.15.63:0/792049006 shutdown_connections 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.460+0000 7f4de4c49640 1 -- 172.21.15.63:0/792049006 wait complete. 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 Processor -- start 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 -- start start 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de01011c0 0x7f4de010df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010eb60 con 0x7f4de0101b60 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4de4c49640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de010ecd0 con 0x7f4de01011c0 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:37802/0 (socket says 172.21.15.63:37802) 2024-06-15T00:07:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 -- 172.21.15.63:0/620423932 learned_addr learned my addr 172.21.15.63:0/620423932 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dde59c640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de01011c0 0x7f4de010df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 -- 172.21.15.63:0/620423932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de01011c0 msgr2=0x7f4de010df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de01011c0 0x7f4de010df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.461+0000 7f4dd7fff640 1 -- 172.21.15.63:0/620423932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0005040 con 0x7f4de0101b60 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4dd7fff640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800b790 tx=0x7f4dc80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dc800c700 con 0x7f4de0101b60 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4dc800cd10 con 0x7f4de0101b60 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de0116fb0 con 0x7f4de0101b60 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dc8013370 con 0x7f4de0101b60 2024-06-15T00:07:43.461 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.462+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de01174a0 con 0x7f4de0101b60 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.463+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de00fc930 con 0x7f4de0101b60 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.463+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4dc8012070 con 0x7f4de0101b60 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.464+0000 7f4dd6ffd640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 0x7f4dcc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.464+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4dc8081fe0 con 0x7f4de0101b60 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.464+0000 7f4dde59c640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 0x7f4dcc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:43.462 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.464+0000 7f4dde59c640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 0x7f4dcc0678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0004540 tx=0x7f4dd0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:43.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.468+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4dc8050510 con 0x7f4de0101b60 2024-06-15T00:07:43.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:43 smithi136 ceph-mon[35883]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:43.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.677+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4de004b480 con 0x7f4de0101b60 2024-06-15T00:07:43.674 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.677+0000 7f4dd6ffd640 1 -- 172.21.15.63:0/620423932 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4dc804ee60 con 0x7f4de0101b60 2024-06-15T00:07:43.675 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:43.675 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:43.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 msgr2=0x7f4dcc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:43.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 0x7f4dcc0678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4dd0004540 tx=0x7f4dd0035040 comp rx=0 tx=0).stop 2024-06-15T00:07:43.676 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 msgr2=0x7f4de010e480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4dc800b790 tx=0x7f4dc80091d0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 shutdown_connections 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4dcc065430 0x7f4dcc0678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f4de0101b60 0x7f4de010e480 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 --2- 172.21.15.63:0/620423932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4de01011c0 0x7f4de010df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 >> 172.21.15.63:0/620423932 conn(0x7f4de00f6d70 msgr2=0x7f4de00f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 shutdown_connections 2024-06-15T00:07:43.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:43.679+0000 7f4de4c49640 1 -- 172.21.15.63:0/620423932 wait complete. 2024-06-15T00:07:44.266 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:44 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/620423932' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:44.433 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:44 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/620423932' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:45.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:45 smithi063 ceph-mon[28882]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:45.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:45 smithi136 ceph-mon[35883]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:47.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:47 smithi063 ceph-mon[28882]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:47.544 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:47 smithi136 ceph-mon[35883]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:07:49.112 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 -- 172.21.15.63:0/2288067634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fe1c0 msgr2=0x7effa80fe590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 --2- 172.21.15.63:0/2288067634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fe1c0 0x7effa80fe590 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7eff98009c50 tx=0x7eff98004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 -- 172.21.15.63:0/2288067634 shutdown_connections 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 --2- 172.21.15.63:0/2288067634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fa940 0x7effa80fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 --2- 172.21.15.63:0/2288067634 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fe1c0 0x7effa80fe590 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 -- 172.21.15.63:0/2288067634 >> 172.21.15.63:0/2288067634 conn(0x7effa8071860 msgr2=0x7effa8071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 -- 172.21.15.63:0/2288067634 shutdown_connections 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.103+0000 7effad3a9640 1 -- 172.21.15.63:0/2288067634 wait complete. 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 Processor -- start 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 -- start start 2024-06-15T00:07:49.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 0x7effa8113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa8113c00 con 0x7effa80fa940 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effad3a9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effa810cfb0 con 0x7effa80fe1c0 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40758/0 (socket says 172.21.15.63:40758) 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 -- 172.21.15.63:0/671827024 learned_addr learned my addr 172.21.15.63:0/671827024 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa67fc640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 0x7effa8113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 -- 172.21.15.63:0/671827024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 msgr2=0x7effa8113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 0x7effa8113570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa6ffd640 1 -- 172.21.15.63:0/671827024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff980098b0 con 0x7effa80fa940 2024-06-15T00:07:49.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.104+0000 7effa67fc640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 0x7effa8113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7effa6ffd640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff98003400 tx=0x7eff98004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff98041070 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff98009690 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effa810d230 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7eff9803d8c0 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.105+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effa810d6f0 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.106+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effa8064ab0 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.106+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7eff9802f030 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.107+0000 7eff9f7fe640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 0x7eff940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.107+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7eff980a30f0 con 0x7effa80fa940 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.107+0000 7effa67fc640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 0x7eff940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:49.115 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.107+0000 7effa67fc640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 0x7eff940678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff90006fd0 tx=0x7eff90008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:49.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.111+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7eff980715a0 con 0x7effa80fa940 2024-06-15T00:07:49.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:49 smithi063 ceph-mon[28882]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:49.320 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.323+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7effa8072370 con 0x7effa80fa940 2024-06-15T00:07:49.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.323+0000 7eff9f7fe640 1 -- 172.21.15.63:0/671827024 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7eff980a1e60 con 0x7effa80fa940 2024-06-15T00:07:49.321 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:49.321 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 msgr2=0x7eff940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 0x7eff940678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7eff90006fd0 tx=0x7eff90008040 comp rx=0 tx=0).stop 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 msgr2=0x7effa8113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7eff98003400 tx=0x7eff98004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 shutdown_connections 2024-06-15T00:07:49.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7eff94065430 0x7eff940678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effa80fe1c0 0x7effa8113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 --2- 172.21.15.63:0/671827024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7effa80fa940 0x7effa8113030 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:49.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 >> 172.21.15.63:0/671827024 conn(0x7effa8071860 msgr2=0x7effa80f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:49.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 shutdown_connections 2024-06-15T00:07:49.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:49.325+0000 7effad3a9640 1 -- 172.21.15.63:0/671827024 wait complete. 2024-06-15T00:07:49.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:49 smithi136 ceph-mon[35883]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:50.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:50 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/671827024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:50.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:50 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/671827024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:51.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:51 smithi063 ceph-mon[28882]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:51.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:51 smithi136 ceph-mon[35883]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:52.894 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:53.172 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:53.396 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:53 smithi063 ceph-mon[28882]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:53.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:53 smithi136 ceph-mon[35883]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- 172.21.15.63:0/1159766600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 msgr2=0x7fbe28102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 --2- 172.21.15.63:0/1159766600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe28102670 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009c50 tx=0x7fbe18004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- 172.21.15.63:0/1159766600 shutdown_connections 2024-06-15T00:07:54.690 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 --2- 172.21.15.63:0/1159766600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 0x7fbe280fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 --2- 172.21.15.63:0/1159766600 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe28102670 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- 172.21.15.63:0/1159766600 >> 172.21.15.63:0/1159766600 conn(0x7fbe280f7e50 msgr2=0x7fbe280fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- 172.21.15.63:0/1159766600 shutdown_connections 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- 172.21.15.63:0/1159766600 wait complete. 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 Processor -- start 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.692+0000 7fbe301cc640 1 -- start start 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe301cc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe301cc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 0x7fbe2806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe301cc640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe28070280 con 0x7fbe281022a0 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe301cc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe280703f0 con 0x7fbe28102c40 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:54.691 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40762/0 (socket says 172.21.15.63:40762) 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 -- 172.21.15.63:0/423531579 learned_addr learned my addr 172.21.15.63:0/423531579 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2d767640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 0x7fbe2806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 -- 172.21.15.63:0/423531579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 msgr2=0x7fbe2806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 0x7fbe2806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 -- 172.21.15.63:0/423531579 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe180098b0 con 0x7fbe281022a0 2024-06-15T00:07:54.692 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe2df68640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe18003400 tx=0x7fbe18004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.693+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe18041070 con 0x7fbe281022a0 2024-06-15T00:07:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.694+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe18009690 con 0x7fbe281022a0 2024-06-15T00:07:54.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.694+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1803d8c0 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.694+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe281192e0 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.694+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe28119660 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.695+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe280fda10 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.695+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fbe1802f030 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.695+0000 7fbe267fc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 0x7fbe100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.695+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbe180a2ff0 con 0x7fbe281022a0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.695+0000 7fbe2d767640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 0x7fbe100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:07:54.694 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.696+0000 7fbe2d767640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 0x7fbe100678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe280710b0 tx=0x7fbe1c009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:07:54.697 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.700+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbe180714a0 con 0x7fbe281022a0 2024-06-15T00:07:54.905 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.908+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe280fba90 con 0x7fbe281022a0 2024-06-15T00:07:54.906 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.909+0000 7fbe267fc640 1 -- 172.21.15.63:0/423531579 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbe18074ac0 con 0x7fbe281022a0 2024-06-15T00:07:54.906 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:07:54.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.910+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 msgr2=0x7fbe100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.910+0000 7fbe301cc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 0x7fbe100678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbe280710b0 tx=0x7fbe1c009120 comp rx=0 tx=0).stop 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.910+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 msgr2=0x7fbe2806f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.910+0000 7fbe301cc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fbe18003400 tx=0x7fbe18004d10 comp rx=0 tx=0).stop 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 shutdown_connections 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fbe10065430 0x7fbe100678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe28102c40 0x7fbe2806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 --2- 172.21.15.63:0/423531579 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fbe281022a0 0x7fbe2806f770 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:07:54.908 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 >> 172.21.15.63:0/423531579 conn(0x7fbe280f7e50 msgr2=0x7fbe280f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:07:54.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 shutdown_connections 2024-06-15T00:07:54.909 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:07:54.911+0000 7fbe301cc640 1 -- 172.21.15.63:0/423531579 wait complete. 2024-06-15T00:07:55.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:55 smithi063 ceph-mon[28882]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:55.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:55 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/423531579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:55.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:55 smithi136 ceph-mon[35883]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:55.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:55 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/423531579' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:07:57.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:57 smithi063 ceph-mon[28882]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:57.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:57 smithi136 ceph-mon[35883]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:58.499 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:07:58.777 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:07:59.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:07:59 smithi063 ceph-mon[28882]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:07:59.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:07:59 smithi136 ceph-mon[35883]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:00.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/1797323793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a80fdea0 msgr2=0x7ff7a80fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:00.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/1797323793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a80fdea0 0x7ff7a80fe310 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff79c0048d0 tx=0x7ff79c009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/1797323793 shutdown_connections 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/1797323793 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a80fdea0 0x7ff7a80fe310 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/1797323793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a8065030 0x7ff7a8065400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/1797323793 >> 172.21.15.63:0/1797323793 conn(0x7ff7a8072110 msgr2=0x7ff7a8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/1797323793 shutdown_connections 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/1797323793 wait complete. 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.357+0000 7ff7ae9c3640 1 Processor -- start 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7ae9c3640 1 -- start start 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7ae9c3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7ae9c3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a80fdea0 0x7ff7a8113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7ae9c3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a8113b30 con 0x7ff7a8065030 2024-06-15T00:08:00.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7ae9c3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7a810cf70 con 0x7ff7a80fdea0 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36274/0 (socket says 172.21.15.63:36274) 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 -- 172.21.15.63:0/4166682918 learned_addr learned my addr 172.21.15.63:0/4166682918 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a77fe640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a80fdea0 0x7ff7a8113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 -- 172.21.15.63:0/4166682918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a80fdea0 msgr2=0x7ff7a8113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a80fdea0 0x7ff7a8113530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 -- 172.21.15.63:0/4166682918 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c005040 con 0x7ff7a8065030 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.358+0000 7ff7a7fff640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff79800eb70 tx=0x7ff79800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.359+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff798007c30 con 0x7ff7a8065030 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.359+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7a810d1f0 con 0x7ff7a8065030 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.359+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff79800f040 con 0x7ff7a8065030 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.359+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff79800ca00 con 0x7ff7a8065030 2024-06-15T00:08:00.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.359+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7a810d710 con 0x7ff7a8065030 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.360+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7a80fda40 con 0x7ff7a8065030 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.360+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff79800cc70 con 0x7ff7a8065030 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.360+0000 7ff7a4ff9640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 0x7ff7900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.361+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff7980820c0 con 0x7ff7a8065030 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.361+0000 7ff7a77fe640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 0x7ff7900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.361+0000 7ff7a77fe640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 0x7ff7900678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff7a810e8b0 tx=0x7ff79c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:00.373 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.364+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff798050620 con 0x7ff7a8065030 2024-06-15T00:08:00.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:00 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:08:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.576+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff7a804b480 con 0x7ff7a8065030 2024-06-15T00:08:00.574 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.577+0000 7ff7a4ff9640 1 -- 172.21.15.63:0/4166682918 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff798053c40 con 0x7ff7a8065030 2024-06-15T00:08:00.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:00.574 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 msgr2=0x7ff7900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 0x7ff7900678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff7a810e8b0 tx=0x7ff79c035040 comp rx=0 tx=0).stop 2024-06-15T00:08:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 msgr2=0x7ff7a8112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:00.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff79800eb70 tx=0x7ff79800b040 comp rx=0 tx=0).stop 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 shutdown_connections 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff790065430 0x7ff7900678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7a80fdea0 0x7ff7a8113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 --2- 172.21.15.63:0/4166682918 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff7a8065030 0x7ff7a8112ff0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 >> 172.21.15.63:0/4166682918 conn(0x7ff7a8072110 msgr2=0x7ff7a8105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 shutdown_connections 2024-06-15T00:08:00.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:00.579+0000 7ff7ae9c3640 1 -- 172.21.15.63:0/4166682918 wait complete. 2024-06-15T00:08:00.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:00 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:08:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:01 smithi063 ceph-mon[28882]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:08:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:08:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:08:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:01 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/4166682918' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:01.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:01 smithi136 ceph-mon[35883]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:01.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:08:01.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:08:01.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:08:01.584 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:01 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/4166682918' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:02.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:03.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:03 smithi063 ceph-mon[28882]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:03.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:03 smithi136 ceph-mon[35883]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:04.168 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:04.444 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:05.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:05 smithi063 ceph-mon[28882]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:05.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:05 smithi136 ceph-mon[35883]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 -- 172.21.15.63:0/1907760009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 msgr2=0x7f2d2c06fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1907760009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06fbe0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0048d0 tx=0x7f2d1c009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 -- 172.21.15.63:0/1907760009 shutdown_connections 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1907760009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c108620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1907760009 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06fbe0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 -- 172.21.15.63:0/1907760009 >> 172.21.15.63:0/1907760009 conn(0x7f2d2c0f7f00 msgr2=0x7f2d2c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 -- 172.21.15.63:0/1907760009 shutdown_connections 2024-06-15T00:08:06.091 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.093+0000 7f2d326d8640 1 -- 172.21.15.63:0/1907760009 wait complete. 2024-06-15T00:08:06.092 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.094+0000 7f2d326d8640 1 Processor -- start 2024-06-15T00:08:06.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.094+0000 7f2d326d8640 1 -- start start 2024-06-15T00:08:06.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.094+0000 7f2d326d8640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:06.093 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d326d8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c06efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d326d8640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c0699c0 con 0x7f2d2c06f810 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d326d8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c069b30 con 0x7f2d2c070120 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36288/0 (socket says 172.21.15.63:36288) 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 -- 172.21.15.63:0/1632160267 learned_addr learned my addr 172.21.15.63:0/1632160267 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2b7fe640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c06efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 -- 172.21.15.63:0/1632160267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 msgr2=0x7f2d2c06efe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c06efe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 -- 172.21.15.63:0/1632160267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d1c005040 con 0x7f2d2c06f810 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2b7fe640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c06efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d2bfff640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0043d0 tx=0x7f2d1c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1c038070 con 0x7f2d2c06f810 2024-06-15T00:08:06.094 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c069db0 con 0x7f2d2c06f810 2024-06-15T00:08:06.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2d1c0096f0 con 0x7f2d2c06f810 2024-06-15T00:08:06.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d1c03d420 con 0x7f2d2c06f810 2024-06-15T00:08:06.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.095+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c06a2d0 con 0x7f2d2c06f810 2024-06-15T00:08:06.095 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.096+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c070f50 con 0x7f2d2c06f810 2024-06-15T00:08:06.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.097+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2d1c030020 con 0x7f2d2c06f810 2024-06-15T00:08:06.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.097+0000 7f2d28ff9640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 0x7f2d140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:06.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.098+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2d1c0a2a00 con 0x7f2d2c06f810 2024-06-15T00:08:06.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.098+0000 7f2d2b7fe640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 0x7f2d140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:06.096 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.098+0000 7f2d2b7fe640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 0x7f2d140678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:06.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.102+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2d1c070eb0 con 0x7f2d2c06f810 2024-06-15T00:08:06.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.314+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2d2c04b480 con 0x7f2d2c06f810 2024-06-15T00:08:06.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.315+0000 7f2d28ff9640 1 -- 172.21.15.63:0/1632160267 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2d1c0744d0 con 0x7f2d2c06f810 2024-06-15T00:08:06.312 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:06.312 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:06.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 msgr2=0x7f2d140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:06.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 0x7f2d140678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2d20007fd0 tx=0x7f2d20009e90 comp rx=0 tx=0).stop 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 msgr2=0x7f2d2c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0043d0 tx=0x7f2d1c009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 shutdown_connections 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2d14065430 0x7f2d140678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d2c070120 0x7f2d2c06efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 --2- 172.21.15.63:0/1632160267 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2d2c06f810 0x7f2d2c06eaa0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 >> 172.21.15.63:0/1632160267 conn(0x7f2d2c0f7f00 msgr2=0x7f2d2c0f9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 shutdown_connections 2024-06-15T00:08:06.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:06.317+0000 7f2d326d8640 1 -- 172.21.15.63:0/1632160267 wait complete. 2024-06-15T00:08:07.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:07 smithi063 ceph-mon[28882]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:07.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:07 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1632160267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:07.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:07 smithi136 ceph-mon[35883]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:07.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:07 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1632160267' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:09.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:09 smithi063 ceph-mon[28882]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:09.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:09 smithi136 ceph-mon[35883]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:10.080 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:10.362 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:11.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:11 smithi063 ceph-mon[28882]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:11.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:11 smithi136 ceph-mon[35883]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.035+0000 7f9da4ffb640 1 -- 172.21.15.63:0/107623489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 msgr2=0x7f9da00fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.035+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/107623489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fcd80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f9d880048d0 tx=0x7f9d88009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 -- 172.21.15.63:0/107623489 shutdown_connections 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/107623489 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fcd80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/107623489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 -- 172.21.15.63:0/107623489 >> 172.21.15.63:0/107623489 conn(0x7f9da0072110 msgr2=0x7f9da0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 -- 172.21.15.63:0/107623489 shutdown_connections 2024-06-15T00:08:12.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 -- 172.21.15.63:0/107623489 wait complete. 2024-06-15T00:08:12.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 Processor -- start 2024-06-15T00:08:12.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.036+0000 7f9da4ffb640 1 -- start start 2024-06-15T00:08:12.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9da4ffb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:12.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9da4ffb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:12.037 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9da4ffb640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00fe9e0 con 0x7f9da00fbe80 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9da4ffb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da00feb50 con 0x7f9da00fb570 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43882/0 (socket says 172.21.15.63:43882) 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 -- 172.21.15.63:0/555617217 learned_addr learned my addr 172.21.15.63:0/555617217 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d9e59c640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:12.038 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 -- 172.21.15.63:0/555617217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 msgr2=0x7f9da00fdf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fdf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 -- 172.21.15.63:0/555617217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d88005040 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d9e59c640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d95d9b640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9d880048a0 tx=0x7f9d88009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d88038070 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d8802cef0 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da00fed80 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.037+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9d88009640 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.038+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da0119590 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.038+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da00fc920 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.039+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9d88030020 con 0x7f9da00fbe80 2024-06-15T00:08:12.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.039+0000 7f9d977fe640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 0x7f9d8c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:12.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.039+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9d880a2860 con 0x7f9da00fbe80 2024-06-15T00:08:12.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.039+0000 7f9d9e59c640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 0x7f9d8c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:12.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.040+0000 7f9d9e59c640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 0x7f9d8c0678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9d90004d80 tx=0x7f9d9000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:12.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.044+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9d88070d90 con 0x7f9da00fbe80 2024-06-15T00:08:12.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.255+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9da0072c20 con 0x7f9da00fbe80 2024-06-15T00:08:12.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.255+0000 7f9d977fe640 1 -- 172.21.15.63:0/555617217 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9d880743b0 con 0x7f9da00fbe80 2024-06-15T00:08:12.253 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:12.253 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:12.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 msgr2=0x7f9d8c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:12.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 0x7f9d8c0678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9d90004d80 tx=0x7f9d9000b040 comp rx=0 tx=0).stop 2024-06-15T00:08:12.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 msgr2=0x7f9da00fe4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:12.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f9d880048a0 tx=0x7f9d88009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 shutdown_connections 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9d8c065430 0x7f9d8c0678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9da00fbe80 0x7f9da00fe4a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 --2- 172.21.15.63:0/555617217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9da00fb570 0x7f9da00fdf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 >> 172.21.15.63:0/555617217 conn(0x7f9da0072110 msgr2=0x7f9da0065570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 shutdown_connections 2024-06-15T00:08:12.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:12.257+0000 7f9da4ffb640 1 -- 172.21.15.63:0/555617217 wait complete. 2024-06-15T00:08:13.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:13 smithi136 ceph-mon[35883]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:13.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:13 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/555617217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:13.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:13 smithi063 ceph-mon[28882]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:13.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:13 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/555617217' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:15.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:15 smithi136 ceph-mon[35883]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:15.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:15 smithi063 ceph-mon[28882]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:15.910 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:16.187 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:17.497 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:17 smithi063 ceph-mon[28882]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:17.497 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:17 smithi136 ceph-mon[35883]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:17.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 -- 172.21.15.63:0/1897252216 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d4102c90 msgr2=0x7fa9d40fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:17.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 --2- 172.21.15.63:0/1897252216 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d4102c90 0x7fa9d40fa9e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa9c00048d0 tx=0x7fa9c0009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 -- 172.21.15.63:0/1897252216 shutdown_connections 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 --2- 172.21.15.63:0/1897252216 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d4102c90 0x7fa9d40fa9e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 --2- 172.21.15.63:0/1897252216 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d41022f0 0x7fa9d41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 -- 172.21.15.63:0/1897252216 >> 172.21.15.63:0/1897252216 conn(0x7fa9d4070a40 msgr2=0x7fa9d4070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 -- 172.21.15.63:0/1897252216 shutdown_connections 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.815+0000 7fa9db19b640 1 -- 172.21.15.63:0/1897252216 wait complete. 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 Processor -- start 2024-06-15T00:08:17.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 -- start start 2024-06-15T00:08:17.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:17.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d4102c90 0x7fa9d41135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:17.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d4113c40 con 0x7fa9d41022f0 2024-06-15T00:08:17.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.816+0000 7fa9db19b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d410cff0 con 0x7fa9d4102c90 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:43896/0 (socket says 172.21.15.63:43896) 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 -- 172.21.15.63:0/382024816 learned_addr learned my addr 172.21.15.63:0/382024816 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d3fff640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d4102c90 0x7fa9d41135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 -- 172.21.15.63:0/382024816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d4102c90 msgr2=0x7fa9d41135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d4102c90 0x7fa9d41135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 -- 172.21.15.63:0/382024816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0005040 con 0x7fa9d41022f0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d8f37640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800b580 tx=0x7fa9c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9c800f570 con 0x7fa9d41022f0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa9c800fb80 con 0x7fa9d41022f0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d410d2d0 con 0x7fa9d41022f0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9c8010d10 con 0x7fa9d41022f0 2024-06-15T00:08:17.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.817+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d410d7f0 con 0x7fa9d41022f0 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.818+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d4065510 con 0x7fa9d41022f0 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.819+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa9c800f6d0 con 0x7fa9d41022f0 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.819+0000 7fa9d17fa640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 0x7fa9c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.819+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa9c80823c0 con 0x7fa9d41022f0 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.819+0000 7fa9d3fff640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 0x7fa9c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:17.818 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.820+0000 7fa9d3fff640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 0x7fa9c40678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9d410e930 tx=0x7fa9c0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:17.820 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:17.823+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa9c80508d0 con 0x7fa9d41022f0 2024-06-15T00:08:18.032 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.035+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa9d404b4b0 con 0x7fa9d41022f0 2024-06-15T00:08:18.033 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.036+0000 7fa9d17fa640 1 -- 172.21.15.63:0/382024816 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa9c8053ef0 con 0x7fa9d41022f0 2024-06-15T00:08:18.033 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:18.033 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 msgr2=0x7fa9c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 0x7fa9c40678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa9d410e930 tx=0x7fa9c0035040 comp rx=0 tx=0).stop 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 msgr2=0x7fa9d4113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800b580 tx=0x7fa9c80091a0 comp rx=0 tx=0).stop 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 shutdown_connections 2024-06-15T00:08:18.035 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa9c4065430 0x7fa9c40678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9d4102c90 0x7fa9d41135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 --2- 172.21.15.63:0/382024816 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa9d41022f0 0x7fa9d4113070 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 >> 172.21.15.63:0/382024816 conn(0x7fa9d4070a40 msgr2=0x7fa9d40f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 shutdown_connections 2024-06-15T00:08:18.036 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:18.038+0000 7fa9db19b640 1 -- 172.21.15.63:0/382024816 wait complete. 2024-06-15T00:08:18.386 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:18 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/382024816' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:18.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:18 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/382024816' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:19.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:19 smithi136 ceph-mon[35883]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:19.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:19 smithi063 ceph-mon[28882]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:21.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:21 smithi136 ceph-mon[35883]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:21.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:21 smithi063 ceph-mon[28882]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:21.745 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:22.022 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:23.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:23 smithi063 ceph-mon[28882]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:23.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:23 smithi136 ceph-mon[35883]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 -- 172.21.15.63:0/1599022953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 msgr2=0x7f443c0fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 --2- 172.21.15.63:0/1599022953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fa9e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f44300048d0 tx=0x7f4430009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 -- 172.21.15.63:0/1599022953 shutdown_connections 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 --2- 172.21.15.63:0/1599022953 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fa9e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 --2- 172.21.15.63:0/1599022953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c1022f0 0x7f443c1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.652 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 -- 172.21.15.63:0/1599022953 >> 172.21.15.63:0/1599022953 conn(0x7f443c070a40 msgr2=0x7f443c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 -- 172.21.15.63:0/1599022953 shutdown_connections 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.653+0000 7f444264f640 1 -- 172.21.15.63:0/1599022953 wait complete. 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 Processor -- start 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 -- start start 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c1022f0 0x7f443c0fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0fda00 con 0x7f443c102c90 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f444264f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c0fdb70 con 0x7f443c1022f0 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53856/0 (socket says 172.21.15.63:53856) 2024-06-15T00:08:23.653 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 -- 172.21.15.63:0/91837683 learned_addr learned my addr 172.21.15.63:0/91837683 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 -- 172.21.15.63:0/91837683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c1022f0 msgr2=0x7f443c0fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c1022f0 0x7f443c0fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.654+0000 7f443b7fe640 1 -- 172.21.15.63:0/91837683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4430005040 con 0x7f443c102c90 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f443b7fe640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f44300048a0 tx=0x7f4430009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4430038070 con 0x7f443c102c90 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c0fc730 con 0x7f443c102c90 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4430032d30 con 0x7f443c102c90 2024-06-15T00:08:23.654 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4430009760 con 0x7f443c102c90 2024-06-15T00:08:23.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.655+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c11dd20 con 0x7f443c102c90 2024-06-15T00:08:23.655 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.656+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c065520 con 0x7f443c102c90 2024-06-15T00:08:23.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.656+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f4430030020 con 0x7f443c102c90 2024-06-15T00:08:23.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.657+0000 7f4438ff9640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 0x7f44240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:23.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.657+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f44300a2a30 con 0x7f443c102c90 2024-06-15T00:08:23.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.657+0000 7f443bfff640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 0x7f44240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:23.656 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.657+0000 7f443bfff640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 0x7f44240678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f442c0097e0 tx=0x7f442c006d20 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:23.658 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.661+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4430070ee0 con 0x7f443c102c90 2024-06-15T00:08:23.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.871+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f443c074960 con 0x7f443c102c90 2024-06-15T00:08:23.868 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.871+0000 7f4438ff9640 1 -- 172.21.15.63:0/91837683 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4430074500 con 0x7f443c102c90 2024-06-15T00:08:23.869 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:23.869 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:23.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 msgr2=0x7f44240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:23.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 0x7f44240678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f442c0097e0 tx=0x7f442c006d20 comp rx=0 tx=0).stop 2024-06-15T00:08:23.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 msgr2=0x7f443c0fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:23.871 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f44300048a0 tx=0x7f4430009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 shutdown_connections 2024-06-15T00:08:23.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4424065430 0x7f44240678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f443c102c90 0x7f443c0fc100 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 --2- 172.21.15.63:0/91837683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c1022f0 0x7f443c0fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:23.872 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 >> 172.21.15.63:0/91837683 conn(0x7f443c070a40 msgr2=0x7f443c0f9540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:23.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 shutdown_connections 2024-06-15T00:08:23.873 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:23.873+0000 7f444264f640 1 -- 172.21.15.63:0/91837683 wait complete. 2024-06-15T00:08:24.474 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:24 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/91837683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:24.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:24 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/91837683' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:25.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:25 smithi136 ceph-mon[35883]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:25.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:25 smithi063 ceph-mon[28882]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:27.501 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:27.531 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:27 smithi063 ceph-mon[28882]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:27.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:27 smithi136 ceph-mon[35883]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:27.779 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:29.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 -- 172.21.15.63:0/1684209820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 msgr2=0x7ffbbc1086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:29.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 --2- 172.21.15.63:0/1684209820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1086f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffba80048d0 tx=0x7ffba8009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.359 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 -- 172.21.15.63:0/1684209820 shutdown_connections 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 --2- 172.21.15.63:0/1684209820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1086f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 --2- 172.21.15.63:0/1684209820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc0fb4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 -- 172.21.15.63:0/1684209820 >> 172.21.15.63:0/1684209820 conn(0x7ffbbc0f6d90 msgr2=0x7ffbbc0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 -- 172.21.15.63:0/1684209820 shutdown_connections 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.343+0000 7ffbc303b640 1 -- 172.21.15.63:0/1684209820 wait complete. 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 Processor -- start 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 -- start start 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc115290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc115dd0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbc303b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc10f210 con 0x7ffbbc0fb110 2024-06-15T00:08:29.360 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbbbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbbbfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48656/0 (socket says 172.21.15.63:48656) 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.344+0000 7ffbbbfff640 1 -- 172.21.15.63:0/373508122 learned_addr learned my addr 172.21.15.63:0/373508122 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbc0dd7640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc115290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbbbfff640 1 -- 172.21.15.63:0/373508122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 msgr2=0x7ffbbc115290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbbbfff640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc115290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbbbfff640 1 -- 172.21.15.63:0/373508122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8005040 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbc0dd7640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbbbfff640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffba8004930 tx=0x7ffba8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba8038070 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc10f490 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba80096f0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ffba8032cd0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.345+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc10f9d0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.361 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.346+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbbc0fa9f0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.347+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ffba8030020 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.347+0000 7ffbb97fa640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 0x7ffbac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.347+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ffba80a2a40 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.347+0000 7ffbc0dd7640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 0x7ffbac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.348+0000 7ffbc0dd7640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 0x7ffbac0678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00041d0 tx=0x7ffbb0009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:29.362 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.351+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ffba8070ef0 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:29 smithi063 ceph-mon[28882]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:29.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.561+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ffbbc04b480 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.561+0000 7ffbb97fa640 1 -- 172.21.15.63:0/373508122 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ffba8074510 con 0x7ffbbc0fbab0 2024-06-15T00:08:29.558 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:29.559 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 msgr2=0x7ffbac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 0x7ffbac0678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffbb00041d0 tx=0x7ffbb0009040 comp rx=0 tx=0).stop 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 msgr2=0x7ffbbc1157d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffba8004930 tx=0x7ffba8009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 shutdown_connections 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ffbac065430 0x7ffbac0678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ffbbc0fbab0 0x7ffbbc1157d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 --2- 172.21.15.63:0/373508122 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffbbc0fb110 0x7ffbbc115290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:29.560 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 >> 172.21.15.63:0/373508122 conn(0x7ffbbc0f6d90 msgr2=0x7ffbbc0f7410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:29.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 shutdown_connections 2024-06-15T00:08:29.561 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:29.563+0000 7ffbc303b640 1 -- 172.21.15.63:0/373508122 wait complete. 2024-06-15T00:08:29.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:29 smithi136 ceph-mon[35883]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:30.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:30 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/373508122' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:30.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:30 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/373508122' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:31.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:31 smithi136 ceph-mon[35883]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:31.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:31 smithi063 ceph-mon[28882]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:32.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:32.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:33.197 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:33.475 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:33.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:33 smithi136 ceph-mon[35883]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:33.596 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:33 smithi063 ceph-mon[28882]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.079+0000 7f04c4ed6640 1 -- 172.21.15.63:0/1755205414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c0071440 msgr2=0x7f04c006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.079+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/1755205414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c0071440 0x7f04c006f810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f04b00048d0 tx=0x7f04b0009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- 172.21.15.63:0/1755205414 shutdown_connections 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/1755205414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c006fd50 0x7f04c00701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/1755205414 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c0071440 0x7f04c006f810 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- 172.21.15.63:0/1755205414 >> 172.21.15.63:0/1755205414 conn(0x7f04c00f7f00 msgr2=0x7f04c00fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:35.080 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- 172.21.15.63:0/1755205414 shutdown_connections 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- 172.21.15.63:0/1755205414 wait complete. 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 Processor -- start 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- start start 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 0x7f04c0119ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c011a190 con 0x7f04c006fd50 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04c4ed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c0113510 con 0x7f04c0071440 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04be59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.080+0000 7f04be59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:48680/0 (socket says 172.21.15.63:48680) 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04be59c640 1 -- 172.21.15.63:0/3882400175 learned_addr learned my addr 172.21.15.63:0/3882400175 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04b7fff640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 0x7f04c0119ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04be59c640 1 -- 172.21.15.63:0/3882400175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 msgr2=0x7f04c0119ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04be59c640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 0x7f04c0119ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.081 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04be59c640 1 -- 172.21.15.63:0/3882400175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b0005040 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04b7fff640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 0x7f04c0119ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04be59c640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04b0032410 tx=0x7f04b0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04b0038070 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c0113740 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04b0032df0 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f04b003de30 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.081+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c0113c00 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.082+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c0070f50 con 0x7f04c006fd50 2024-06-15T00:08:35.082 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.083+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f04b0030020 con 0x7f04c006fd50 2024-06-15T00:08:35.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.083+0000 7f04b6ffd640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 0x7f04ac067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:35.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.084+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f04b00a30f0 con 0x7f04c006fd50 2024-06-15T00:08:35.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.084+0000 7f04b7fff640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 0x7f04ac067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:35.083 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.084+0000 7f04b7fff640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 0x7f04ac067880 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04a8006fd0 tx=0x7f04a8008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:35.084 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.088+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f04b0071650 con 0x7f04c006fd50 2024-06-15T00:08:35.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.297+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f04c010f960 con 0x7f04c006fd50 2024-06-15T00:08:35.294 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.297+0000 7f04b6ffd640 1 -- 172.21.15.63:0/3882400175 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f04b00a1e60 con 0x7f04c006fd50 2024-06-15T00:08:35.295 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:35.295 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:35.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 msgr2=0x7f04ac067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:35.296 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 0x7f04ac067880 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f04a8006fd0 tx=0x7f04a8008040 comp rx=0 tx=0).stop 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 msgr2=0x7f04c0119570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f04b0032410 tx=0x7f04b0009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 shutdown_connections 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f04ac0653d0 0x7f04ac067880 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c0071440 0x7f04c0119ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 --2- 172.21.15.63:0/3882400175 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f04c006fd50 0x7f04c0119570 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 >> 172.21.15.63:0/3882400175 conn(0x7f04c00f7f00 msgr2=0x7f04c00f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 shutdown_connections 2024-06-15T00:08:35.297 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:35.299+0000 7f04c4ed6640 1 -- 172.21.15.63:0/3882400175 wait complete. 2024-06-15T00:08:35.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:35 smithi136 ceph-mon[35883]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:35.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:35 smithi063 ceph-mon[28882]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:36.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:36 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3882400175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:36.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:36 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3882400175' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:37.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:37 smithi136 ceph-mon[35883]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:37.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:37 smithi063 ceph-mon[28882]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:39.072 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:39.345 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:39.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:39 smithi063 ceph-mon[28882]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:39.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:39 smithi136 ceph-mon[35883]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:40.568 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:40 smithi063 ceph-mon[28882]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:40.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:40 smithi136 ceph-mon[35883]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:40.891 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- 172.21.15.63:0/714758991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc101b70 msgr2=0x7f88fc0fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 --2- 172.21.15.63:0/714758991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc101b70 0x7f88fc0fb200 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f88f80048d0 tx=0x7f88f8009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- 172.21.15.63:0/714758991 shutdown_connections 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 --2- 172.21.15.63:0/714758991 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc101b70 0x7f88fc0fb200 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 --2- 172.21.15.63:0/714758991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc1011d0 0x7f88fc1015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- 172.21.15.63:0/714758991 >> 172.21.15.63:0/714758991 conn(0x7f88fc0f6dc0 msgr2=0x7f88fc0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- 172.21.15.63:0/714758991 shutdown_connections 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- 172.21.15.63:0/714758991 wait complete. 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 Processor -- start 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- start start 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:40.892 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 0x7f88fc1134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc113ab0 con 0x7f88fc1011d0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.893+0000 7f8903302640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88fc10cef0 con 0x7f88fc101b70 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50502/0 (socket says 172.21.15.63:50502) 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 -- 172.21.15.63:0/2475447188 learned_addr learned my addr 172.21.15.63:0/2475447188 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890089d640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 0x7f88fc1134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 -- 172.21.15.63:0/2475447188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 msgr2=0x7f88fc1134b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 0x7f88fc1134b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 -- 172.21.15.63:0/2475447188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8005040 con 0x7f88fc1011d0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890089d640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 0x7f88fc1134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f890109e640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c830 tx=0x7f88f0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f000a850 con 0x7f88fc1011d0 2024-06-15T00:08:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88fc10d1d0 con 0x7f88fc1011d0 2024-06-15T00:08:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88f000ae60 con 0x7f88fc1011d0 2024-06-15T00:08:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f88f00113a0 con 0x7f88fc1011d0 2024-06-15T00:08:40.894 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.894+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88fc10d6f0 con 0x7f88fc1011d0 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.895+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88fc0fc940 con 0x7f88fc1011d0 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.896+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f88f0012400 con 0x7f88fc1011d0 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.896+0000 7f88edffb640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 0x7f88e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.896+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f88f00826e0 con 0x7f88fc1011d0 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.896+0000 7f890089d640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 0x7f88e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:40.895 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.897+0000 7f890089d640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 0x7f88e80678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88fc10e830 tx=0x7f88f8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:40.897 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:40.900+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f88f0050cd0 con 0x7f88fc1011d0 2024-06-15T00:08:41.113 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.116+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f88fc0fa9c0 con 0x7f88fc1011d0 2024-06-15T00:08:41.114 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.116+0000 7f88edffb640 1 -- 172.21.15.63:0/2475447188 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f88f00542f0 con 0x7f88fc1011d0 2024-06-15T00:08:41.114 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:41.114 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 msgr2=0x7f88e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 0x7f88e80678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f88fc10e830 tx=0x7f88f8035040 comp rx=0 tx=0).stop 2024-06-15T00:08:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 msgr2=0x7f88fc112f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:41.116 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f88f000c830 tx=0x7f88f0004430 comp rx=0 tx=0).stop 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 shutdown_connections 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f88e8065430 0x7f88e80678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88fc101b70 0x7f88fc1134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 --2- 172.21.15.63:0/2475447188 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f88fc1011d0 0x7f88fc112f70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 >> 172.21.15.63:0/2475447188 conn(0x7f88fc0f6dc0 msgr2=0x7f88fc0f84d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 shutdown_connections 2024-06-15T00:08:41.117 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:41.119+0000 7f8903302640 1 -- 172.21.15.63:0/2475447188 wait complete. 2024-06-15T00:08:41.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:41 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2475447188' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:41.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:41 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2475447188' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:42.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:42 smithi136 ceph-mon[35883]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:42.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:42 smithi063 ceph-mon[28882]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:44.767 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:45.045 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:45.168 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:44 smithi063 ceph-mon[28882]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:45.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:44 smithi136 ceph-mon[35883]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- 172.21.15.63:0/1394804686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fb570 msgr2=0x7fc8100fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 --2- 172.21.15.63:0/1394804686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fb570 0x7fc8100fb940 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc800009c50 tx=0x7fc800004d10 comp rx=0 tx=0).stop 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- 172.21.15.63:0/1394804686 shutdown_connections 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 --2- 172.21.15.63:0/1394804686 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fbe80 0x7fc8100fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 --2- 172.21.15.63:0/1394804686 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fb570 0x7fc8100fb940 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- 172.21.15.63:0/1394804686 >> 172.21.15.63:0/1394804686 conn(0x7fc810072110 msgr2=0x7fc810072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- 172.21.15.63:0/1394804686 shutdown_connections 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- 172.21.15.63:0/1394804686 wait complete. 2024-06-15T00:08:46.621 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 Processor -- start 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.622+0000 7fc815d89640 1 -- start start 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc815d89640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fb570 0x7fc81010df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc815d89640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc815d89640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81010eb80 con 0x7fc8100fbe80 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc815d89640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81010ecf0 con 0x7fc8100fb570 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:50524/0 (socket says 172.21.15.63:50524) 2024-06-15T00:08:46.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 -- 172.21.15.63:0/1782609694 learned_addr learned my addr 172.21.15.63:0/1782609694 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:46.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80f7fe640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fb570 0x7fc81010df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 -- 172.21.15.63:0/1782609694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fb570 msgr2=0x7fc81010df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fb570 0x7fc81010df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 -- 172.21.15.63:0/1782609694 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8000098b0 con 0x7fc8100fbe80 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.623+0000 7fc80effd640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc80400b790 tx=0x7fc8040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.624+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc80400c700 con 0x7fc8100fbe80 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.624+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc810116fd0 con 0x7fc8100fbe80 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.624+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc80400cd10 con 0x7fc8100fbe80 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.624+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc804013370 con 0x7fc8100fbe80 2024-06-15T00:08:46.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.624+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8101174c0 con 0x7fc8100fbe80 2024-06-15T00:08:46.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.625+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8100fc920 con 0x7fc8100fbe80 2024-06-15T00:08:46.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.625+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc804012070 con 0x7fc8100fbe80 2024-06-15T00:08:46.627 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.626+0000 7fc7f7fff640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 0x7fc7f80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:46.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.626+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc804082090 con 0x7fc8100fbe80 2024-06-15T00:08:46.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.626+0000 7fc80f7fe640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 0x7fc7f80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:46.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.626+0000 7fc80f7fe640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 0x7fc7f80678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc800003400 tx=0x7fc800004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:46.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.630+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc804050540 con 0x7fc8100fbe80 2024-06-15T00:08:46.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.839+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc810072c20 con 0x7fc8100fbe80 2024-06-15T00:08:46.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.839+0000 7fc7f7fff640 1 -- 172.21.15.63:0/1782609694 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc804053b60 con 0x7fc8100fbe80 2024-06-15T00:08:46.838 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:46.838 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:46.838 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 msgr2=0x7fc7f80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 0x7fc7f80678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fc800003400 tx=0x7fc800004d10 comp rx=0 tx=0).stop 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 msgr2=0x7fc81010e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc80400b790 tx=0x7fc8040091d0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 shutdown_connections 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc7f8065430 0x7fc7f80678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8100fbe80 0x7fc81010e4a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 --2- 172.21.15.63:0/1782609694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8100fb570 0x7fc81010df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 >> 172.21.15.63:0/1782609694 conn(0x7fc810072110 msgr2=0x7fc8100f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 shutdown_connections 2024-06-15T00:08:46.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:46.841+0000 7fc815d89640 1 -- 172.21.15.63:0/1782609694 wait complete. 2024-06-15T00:08:47.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:46 smithi063 ceph-mon[28882]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:47.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:46 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1782609694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:47.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:46 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:47.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:46 smithi136 ceph-mon[35883]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:47.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:46 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1782609694' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:47.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:46 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:08:49.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:48 smithi136 ceph-mon[35883]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:49.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:48 smithi063 ceph-mon[28882]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:50.487 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:50.762 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:51.175 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:50 smithi063 ceph-mon[28882]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:51.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:50 smithi136 ceph-mon[35883]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:52.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.352+0000 7fa1e098d640 1 -- 172.21.15.63:0/2718882037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc102c60 msgr2=0x7fa1dc0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:52.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.352+0000 7fa1e098d640 1 --2- 172.21.15.63:0/2718882037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc102c60 0x7fa1dc0fc2f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00048d0 tx=0x7fa1d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.369 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- 172.21.15.63:0/2718882037 shutdown_connections 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 --2- 172.21.15.63:0/2718882037 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc102c60 0x7fa1dc0fc2f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 --2- 172.21.15.63:0/2718882037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- 172.21.15.63:0/2718882037 >> 172.21.15.63:0/2718882037 conn(0x7fa1dc0f7e50 msgr2=0x7fa1dc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- 172.21.15.63:0/2718882037 shutdown_connections 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- 172.21.15.63:0/2718882037 wait complete. 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 Processor -- start 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- start start 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc102c60 0x7fa1dc1137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc113e70 con 0x7fa1dc1022c0 2024-06-15T00:08:52.370 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1e098d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1dc10d220 con 0x7fa1dc102c60 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.353+0000 7fa1da59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59556/0 (socket says 172.21.15.63:59556) 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 -- 172.21.15.63:0/46428662 learned_addr learned my addr 172.21.15.63:0/46428662 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1d9d9b640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc102c60 0x7fa1dc1137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 -- 172.21.15.63:0/46428662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc102c60 msgr2=0x7fa1dc1137e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc102c60 0x7fa1dc1137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 -- 172.21.15.63:0/46428662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0005040 con 0x7fa1dc1022c0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1da59c640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00b790 tx=0x7fa1cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1cc00c700 con 0x7fa1dc1022c0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1dc10d500 con 0x7fa1dc1022c0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1cc00cd10 con 0x7fa1dc1022c0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa1cc013370 con 0x7fa1dc1022c0 2024-06-15T00:08:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.354+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1dc10da20 con 0x7fa1dc1022c0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.355+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1dc0fda30 con 0x7fa1dc1022c0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.356+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fa1cc012070 con 0x7fa1dc1022c0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.356+0000 7fa1caffd640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 0x7fa1c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.356+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa1cc082170 con 0x7fa1dc1022c0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.356+0000 7fa1d9d9b640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 0x7fa1c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.357+0000 7fa1d9d9b640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 0x7fa1c40678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0003080 tx=0x7fa1d0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:52.372 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.360+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa1cc050620 con 0x7fa1dc1022c0 2024-06-15T00:08:52.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.576+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa1dc04b4e0 con 0x7fa1dc1022c0 2024-06-15T00:08:52.573 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.576+0000 7fa1caffd640 1 -- 172.21.15.63:0/46428662 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa1cc053c40 con 0x7fa1dc1022c0 2024-06-15T00:08:52.574 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:52.574 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 msgr2=0x7fa1c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 0x7fa1c40678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0003080 tx=0x7fa1d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 msgr2=0x7fa1dc1132a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00b790 tx=0x7fa1cc0091d0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 shutdown_connections 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fa1c4065430 0x7fa1c40678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1dc102c60 0x7fa1dc1137e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 --2- 172.21.15.63:0/46428662 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fa1dc1022c0 0x7fa1dc1132a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 >> 172.21.15.63:0/46428662 conn(0x7fa1dc0f7e50 msgr2=0x7fa1dc0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 shutdown_connections 2024-06-15T00:08:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:52.578+0000 7fa1e098d640 1 -- 172.21.15.63:0/46428662 wait complete. 2024-06-15T00:08:53.139 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:52 smithi063 ceph-mon[28882]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:53.140 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:52 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/46428662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:53.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:52 smithi136 ceph-mon[35883]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:53.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:52 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/46428662' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:55.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:54 smithi136 ceph-mon[35883]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:55.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:54 smithi063 ceph-mon[28882]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:56.166 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:08:56.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:08:57.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:56 smithi063 ceph-mon[28882]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:57.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:56 smithi136 ceph-mon[35883]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:58.039 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 -- 172.21.15.63:0/2609304174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fb570 msgr2=0x7f6c4c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 --2- 172.21.15.63:0/2609304174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c0fb940 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c0048d0 tx=0x7f6c3c009da0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 -- 172.21.15.63:0/2609304174 shutdown_connections 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 --2- 172.21.15.63:0/2609304174 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c0fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 --2- 172.21.15.63:0/2609304174 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c0fb940 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 -- 172.21.15.63:0/2609304174 >> 172.21.15.63:0/2609304174 conn(0x7f6c4c072110 msgr2=0x7f6c4c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 -- 172.21.15.63:0/2609304174 shutdown_connections 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.040+0000 7f6c52456640 1 -- 172.21.15.63:0/2609304174 wait complete. 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 Processor -- start 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 -- start start 2024-06-15T00:08:58.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c113a50 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c52456640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c10ce90 con 0x7f6c4c0fb570 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c4b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c4b7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59574/0 (socket says 172.21.15.63:59574) 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.041+0000 7f6c4b7fe640 1 -- 172.21.15.63:0/3363996869 learned_addr learned my addr 172.21.15.63:0/3363996869 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c4bfff640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c4b7fe640 1 -- 172.21.15.63:0/3363996869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fb570 msgr2=0x7f6c4c112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c4b7fe640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c4b7fe640 1 -- 172.21.15.63:0/3363996869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c005040 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c4b7fe640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6c40002a50 tx=0x7f6c40004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:58.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c40007c30 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c4000f040 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c10d170 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c4000da10 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.042+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c10d710 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.043+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c0fc920 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.044+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6c4000dc90 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.044+0000 7f6c48ff9640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 0x7f6c340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.044+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6c400823e0 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.044+0000 7f6c4bfff640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 0x7f6c340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:08:58.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.045+0000 7f6c4bfff640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 0x7f6c340678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c003080 tx=0x7f6c3c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:08:58.045 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.048+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6c40050890 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.256+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6c4c072c20 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.256+0000 7f6c48ff9640 1 -- 172.21.15.63:0/3363996869 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6c40053eb0 con 0x7f6c4c0fbe80 2024-06-15T00:08:58.254 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:08:58.254 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:08:58.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 msgr2=0x7f6c340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:58.255 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 0x7f6c340678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c003080 tx=0x7f6c3c035040 comp rx=0 tx=0).stop 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 msgr2=0x7f6c4c113450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6c40002a50 tx=0x7f6c40004600 comp rx=0 tx=0).stop 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 shutdown_connections 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6c34065430 0x7f6c340678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6c4c0fbe80 0x7f6c4c113450 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 --2- 172.21.15.63:0/3363996869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6c4c0fb570 0x7f6c4c112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 >> 172.21.15.63:0/3363996869 conn(0x7f6c4c072110 msgr2=0x7f6c4c0fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:08:58.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 shutdown_connections 2024-06-15T00:08:58.257 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:08:58.258+0000 7f6c52456640 1 -- 172.21.15.63:0/3363996869 wait complete. 2024-06-15T00:08:59.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:58 smithi063 ceph-mon[28882]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:59.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:08:58 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3363996869' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:08:59.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:58 smithi136 ceph-mon[35883]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:08:59.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:08:58 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3363996869' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:01 smithi136 ceph-mon[35883]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:09:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:09:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:09:01.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:09:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:01 smithi063 ceph-mon[28882]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:09:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:09:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:09:01.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:09:01.871 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:02.151 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:02.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:02.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:03.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:03 smithi063 ceph-mon[28882]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:03.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:03 smithi136 ceph-mon[35883]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 -- 172.21.15.63:0/64559024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c101b50 msgr2=0x7ff66c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 --2- 172.21.15.63:0/64559024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c101b50 0x7ff66c0fb1e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff6600048d0 tx=0x7ff660009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 -- 172.21.15.63:0/64559024 shutdown_connections 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 --2- 172.21.15.63:0/64559024 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c101b50 0x7ff66c0fb1e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 --2- 172.21.15.63:0/64559024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c1011b0 0x7ff66c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 -- 172.21.15.63:0/64559024 >> 172.21.15.63:0/64559024 conn(0x7ff66c0f6d80 msgr2=0x7ff66c0f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 -- 172.21.15.63:0/64559024 shutdown_connections 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 -- 172.21.15.63:0/64559024 wait complete. 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.811+0000 7ff6725bd640 1 Processor -- start 2024-06-15T00:09:03.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff6725bd640 1 -- start start 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff6725bd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff6725bd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c101b50 0x7ff66c10e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff6725bd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c10ea90 con 0x7ff66c1011b0 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff6725bd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66c10ec00 con 0x7ff66c101b50 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52970/0 (socket says 172.21.15.63:52970) 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 -- 172.21.15.63:0/1438604679 learned_addr learned my addr 172.21.15.63:0/1438604679 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66b7fe640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c101b50 0x7ff66c10e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 -- 172.21.15.63:0/1438604679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c101b50 msgr2=0x7ff66c10e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c101b50 0x7ff66c10e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 -- 172.21.15.63:0/1438604679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff660005040 con 0x7ff66c1011b0 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.812+0000 7ff66bfff640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c790 tx=0x7ff65c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:03.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.813+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c00a820 con 0x7ff66c1011b0 2024-06-15T00:09:03.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.813+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff66c116f70 con 0x7ff66c1011b0 2024-06-15T00:09:03.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.813+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff65c00ae30 con 0x7ff66c1011b0 2024-06-15T00:09:03.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.813+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff65c0113a0 con 0x7ff66c1011b0 2024-06-15T00:09:03.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.813+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66c117480 con 0x7ff66c1011b0 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.814+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff66c0fc920 con 0x7ff66c1011b0 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.814+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff65c00a980 con 0x7ff66c1011b0 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.815+0000 7ff668ff9640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 0x7ff6540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.815+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff65c082630 con 0x7ff66c1011b0 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.815+0000 7ff66b7fe640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 0x7ff6540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:03.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.815+0000 7ff66b7fe640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 0x7ff6540678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff660004540 tx=0x7ff660035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:03.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:03.819+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff65c050b60 con 0x7ff66c1011b0 2024-06-15T00:09:04.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.028+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff66c0fa9a0 con 0x7ff66c1011b0 2024-06-15T00:09:04.026 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.028+0000 7ff668ff9640 1 -- 172.21.15.63:0/1438604679 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff65c054180 con 0x7ff66c1011b0 2024-06-15T00:09:04.026 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:04.026 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 msgr2=0x7ff6540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 0x7ff6540678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff660004540 tx=0x7ff660035040 comp rx=0 tx=0).stop 2024-06-15T00:09:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 msgr2=0x7ff66c10df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff65c00c790 tx=0x7ff65c0043c0 comp rx=0 tx=0).stop 2024-06-15T00:09:04.028 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 shutdown_connections 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff654065430 0x7ff6540678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff66c101b50 0x7ff66c10e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 --2- 172.21.15.63:0/1438604679 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff66c1011b0 0x7ff66c10df00 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 >> 172.21.15.63:0/1438604679 conn(0x7ff66c0f6d80 msgr2=0x7ff66c0f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 shutdown_connections 2024-06-15T00:09:04.029 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:04.031+0000 7ff6725bd640 1 -- 172.21.15.63:0/1438604679 wait complete. 2024-06-15T00:09:05.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:05 smithi136 ceph-mon[35883]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:05.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:05 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1438604679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:05.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:05 smithi063 ceph-mon[28882]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:05.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:05 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1438604679' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:07.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:07 smithi136 ceph-mon[35883]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:07.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:07 smithi063 ceph-mon[28882]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:07.734 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:08.011 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:09.323 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:09 smithi063 ceph-mon[28882]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:09.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:09 smithi136 ceph-mon[35883]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:09.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 -- 172.21.15.63:0/1091005820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f80700c0 msgr2=0x7f08f8108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:09.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/1091005820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f80700c0 0x7f08f8108630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f08e80048d0 tx=0x7f08e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 -- 172.21.15.63:0/1091005820 shutdown_connections 2024-06-15T00:09:09.622 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/1091005820 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f80700c0 0x7f08f8108630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/1091005820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f806f7b0 0x7f08f806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 -- 172.21.15.63:0/1091005820 >> 172.21.15.63:0/1091005820 conn(0x7f08f80f7e50 msgr2=0x7f08f80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 -- 172.21.15.63:0/1091005820 shutdown_connections 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.623+0000 7f08ff8fa640 1 -- 172.21.15.63:0/1091005820 wait complete. 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 Processor -- start 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 -- start start 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80700c0 0x7f08f81156f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8115cf0 con 0x7f08f806f7b0 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08ff8fa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f810f130 con 0x7f08f80700c0 2024-06-15T00:09:09.623 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.624+0000 7f08fd696640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36150/0 (socket says 172.21.15.63:36150) 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 -- 172.21.15.63:0/265636538 learned_addr learned my addr 172.21.15.63:0/265636538 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 -- 172.21.15.63:0/265636538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80700c0 msgr2=0x7f08f81156f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80700c0 0x7f08f81156f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 -- 172.21.15.63:0/265636538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8005040 con 0x7f08f806f7b0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08fd696640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08f400c790 tx=0x7f08f40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08f400a820 con 0x7f08f806f7b0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08f400ae30 con 0x7f08f806f7b0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f810f410 con 0x7f08f806f7b0 2024-06-15T00:09:09.624 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f08f40113a0 con 0x7f08f806f7b0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.625+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f810f930 con 0x7f08f806f7b0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.626+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f8070ef0 con 0x7f08f806f7b0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.627+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f08f4012400 con 0x7f08f806f7b0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.627+0000 7f08edffb640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 0x7f08e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.627+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f08f40826b0 con 0x7f08f806f7b0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.627+0000 7f08fce95640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 0x7f08e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:09.626 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.628+0000 7f08fce95640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 0x7f08e40678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08e8003080 tx=0x7f08e8009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:09.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.631+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f08f4050be0 con 0x7f08f806f7b0 2024-06-15T00:09:09.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.840+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08f8110820 con 0x7f08f806f7b0 2024-06-15T00:09:09.837 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.840+0000 7f08edffb640 1 -- 172.21.15.63:0/265636538 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f08f400a980 con 0x7f08f806f7b0 2024-06-15T00:09:09.837 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:09.838 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:09.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 msgr2=0x7f08e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:09.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 0x7f08e40678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f08e8003080 tx=0x7f08e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 msgr2=0x7f08f81151b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f08f400c790 tx=0x7f08f40043c0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 shutdown_connections 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f08e4065430 0x7f08e40678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08f80700c0 0x7f08f81156f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 --2- 172.21.15.63:0/265636538 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f08f806f7b0 0x7f08f81151b0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 >> 172.21.15.63:0/265636538 conn(0x7f08f80f7e50 msgr2=0x7f08f80f9380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 shutdown_connections 2024-06-15T00:09:09.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:09.842+0000 7f08ff8fa640 1 -- 172.21.15.63:0/265636538 wait complete. 2024-06-15T00:09:10.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:10 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/265636538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:10.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:10 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/265636538' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:11.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:11 smithi136 ceph-mon[35883]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:11.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:11 smithi063 ceph-mon[28882]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:13.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:13 smithi136 ceph-mon[35883]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:13.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:13 smithi063 ceph-mon[28882]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:13.592 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:13.870 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:15.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:15 smithi136 ceph-mon[35883]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:15.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:15 smithi063 ceph-mon[28882]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 -- 172.21.15.63:0/132665558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c701011b0 msgr2=0x7f9c70101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 --2- 172.21.15.63:0/132665558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c701011b0 0x7f9c70101580 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0048d0 tx=0x7f9c6c009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 -- 172.21.15.63:0/132665558 shutdown_connections 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 --2- 172.21.15.63:0/132665558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c70101b50 0x7f9c700fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 --2- 172.21.15.63:0/132665558 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c701011b0 0x7f9c70101580 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 -- 172.21.15.63:0/132665558 >> 172.21.15.63:0/132665558 conn(0x7f9c700f6d80 msgr2=0x7f9c700f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 -- 172.21.15.63:0/132665558 shutdown_connections 2024-06-15T00:09:15.563 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.564+0000 7f9c77e24640 1 -- 172.21.15.63:0/132665558 wait complete. 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 Processor -- start 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 -- start start 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c701011b0 0x7f9c7010df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7010ebb0 con 0x7f9c70101b50 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c77e24640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7010ed20 con 0x7f9c701011b0 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c753bf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c753bf640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:36176/0 (socket says 172.21.15.63:36176) 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c753bf640 1 -- 172.21.15.63:0/3022661866 learned_addr learned my addr 172.21.15.63:0/3022661866 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.565+0000 7f9c75bc0640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c701011b0 0x7f9c7010df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c753bf640 1 -- 172.21.15.63:0/3022661866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c701011b0 msgr2=0x7f9c7010df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c753bf640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c701011b0 0x7f9c7010df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c753bf640 1 -- 172.21.15.63:0/3022661866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6c005040 con 0x7f9c70101b50 2024-06-15T00:09:15.564 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c753bf640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000b790 tx=0x7f9c600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:15.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c6000c700 con 0x7f9c70101b50 2024-06-15T00:09:15.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c70117000 con 0x7f9c70101b50 2024-06-15T00:09:15.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c6000cd10 con 0x7f9c70101b50 2024-06-15T00:09:15.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c60013370 con 0x7f9c70101b50 2024-06-15T00:09:15.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.566+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c70117520 con 0x7f9c70101b50 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.567+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c700fc920 con 0x7f9c70101b50 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.567+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9c60012070 con 0x7f9c70101b50 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.568+0000 7f9c667fc640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 0x7f9c5c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.568+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c60081f40 con 0x7f9c70101b50 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.568+0000 7f9c75bc0640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 0x7f9c5c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:15.566 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.568+0000 7f9c75bc0640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 0x7f9c5c0678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c007ac0 tx=0x7f9c6c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:15.569 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.572+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c600503f0 con 0x7f9c70101b50 2024-06-15T00:09:15.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.785+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c7004b480 con 0x7f9c70101b50 2024-06-15T00:09:15.782 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.785+0000 7f9c667fc640 1 -- 172.21.15.63:0/3022661866 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9c60053a10 con 0x7f9c70101b50 2024-06-15T00:09:15.782 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:15.783 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 msgr2=0x7f9c5c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 0x7f9c5c0678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c007ac0 tx=0x7f9c6c035040 comp rx=0 tx=0).stop 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 msgr2=0x7f9c7010e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9c6000b790 tx=0x7f9c600091d0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 shutdown_connections 2024-06-15T00:09:15.784 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9c5c065430 0x7f9c5c0678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9c70101b50 0x7f9c7010e4d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 --2- 172.21.15.63:0/3022661866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c701011b0 0x7f9c7010df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:15.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 >> 172.21.15.63:0/3022661866 conn(0x7f9c700f6d80 msgr2=0x7f9c700f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:15.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 shutdown_connections 2024-06-15T00:09:15.785 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:15.787+0000 7f9c77e24640 1 -- 172.21.15.63:0/3022661866 wait complete. 2024-06-15T00:09:16.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:16 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3022661866' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:16.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:16 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3022661866' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:17.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:17 smithi136 ceph-mon[35883]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:17.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:17 smithi063 ceph-mon[28882]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:19.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:19 smithi136 ceph-mon[35883]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:19.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:19 smithi063 ceph-mon[28882]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:19.505 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:19.783 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:21.253 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:21 smithi063 ceph-mon[28882]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:21.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:21 smithi136 ceph-mon[35883]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:21.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 -- 172.21.15.63:0/2896471195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d81022a0 msgr2=0x7f62d8102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:21.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 --2- 172.21.15.63:0/2896471195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d81022a0 0x7f62d8102670 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f62c80048d0 tx=0x7f62c8009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 -- 172.21.15.63:0/2896471195 shutdown_connections 2024-06-15T00:09:21.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 --2- 172.21.15.63:0/2896471195 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d8102c40 0x7f62d80fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.463 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 --2- 172.21.15.63:0/2896471195 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d81022a0 0x7f62d8102670 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 -- 172.21.15.63:0/2896471195 >> 172.21.15.63:0/2896471195 conn(0x7f62d80f7e50 msgr2=0x7f62d80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 -- 172.21.15.63:0/2896471195 shutdown_connections 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 -- 172.21.15.63:0/2896471195 wait complete. 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.463+0000 7f62de638640 1 Processor -- start 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62de638640 1 -- start start 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62de638640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d81022a0 0x7f62d8112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62de638640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62de638640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d8113a20 con 0x7f62d8102c40 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62de638640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d810ce60 con 0x7f62d81022a0 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52362/0 (socket says 172.21.15.63:52362) 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 -- 172.21.15.63:0/1744972051 learned_addr learned my addr 172.21.15.63:0/1744972051 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:21.464 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d7fff640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d81022a0 0x7f62d8112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 -- 172.21.15.63:0/1744972051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d81022a0 msgr2=0x7f62d8112ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d81022a0 0x7f62d8112ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 -- 172.21.15.63:0/1744972051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8005040 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.464+0000 7f62d77fe640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c810 tx=0x7f62cc0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.465+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc00a820 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.465+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62cc00ae30 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.465+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d810d140 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.465+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f62cc0115c0 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.465+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d810d6e0 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.466+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d80fda10 con 0x7f62d8102c40 2024-06-15T00:09:21.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.466+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f62cc00a980 con 0x7f62d8102c40 2024-06-15T00:09:21.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.467+0000 7f62d4ff9640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 0x7f62c00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:21.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.467+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f62cc0829c0 con 0x7f62d8102c40 2024-06-15T00:09:21.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.467+0000 7f62d7fff640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 0x7f62c00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:21.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.467+0000 7f62d7fff640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 0x7f62c00678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62c8004540 tx=0x7f62c8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:21.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.471+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f62cc050ef0 con 0x7f62d8102c40 2024-06-15T00:09:21.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.680+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f62d810e550 con 0x7f62d8102c40 2024-06-15T00:09:21.677 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.680+0000 7f62d4ff9640 1 -- 172.21.15.63:0/1744972051 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f62cc054510 con 0x7f62d8102c40 2024-06-15T00:09:21.678 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:21.678 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:21.679 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 msgr2=0x7f62c00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 0x7f62c00678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f62c8004540 tx=0x7f62c8035040 comp rx=0 tx=0).stop 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 msgr2=0x7f62d8113420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f62cc00c810 tx=0x7f62cc0043c0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 shutdown_connections 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f62c0065430 0x7f62c00678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f62d8102c40 0x7f62d8113420 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 --2- 172.21.15.63:0/1744972051 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62d81022a0 0x7f62d8112ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.682+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 >> 172.21.15.63:0/1744972051 conn(0x7f62d80f7e50 msgr2=0x7f62d80f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.683+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 shutdown_connections 2024-06-15T00:09:21.680 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:21.683+0000 7f62de638640 1 -- 172.21.15.63:0/1744972051 wait complete. 2024-06-15T00:09:22.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:22 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1744972051' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:22.391 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:22 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1744972051' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:23.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:23 smithi136 ceph-mon[35883]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:23.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:23 smithi063 ceph-mon[28882]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:25.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:25 smithi136 ceph-mon[35883]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:25.421 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:25.430 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:25 smithi063 ceph-mon[28882]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:25.699 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:27.156 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:27 smithi063 ceph-mon[28882]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:27.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:27 smithi136 ceph-mon[35883]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:27.347 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 -- 172.21.15.63:0/340538741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 msgr2=0x7f45a8065110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 --2- 172.21.15.63:0/340538741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a8065110 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f459c0048d0 tx=0x7f459c009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 -- 172.21.15.63:0/340538741 shutdown_connections 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 --2- 172.21.15.63:0/340538741 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 --2- 172.21.15.63:0/340538741 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a8065110 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.350+0000 7f45aed25640 1 -- 172.21.15.63:0/340538741 >> 172.21.15.63:0/340538741 conn(0x7f45a8072110 msgr2=0x7f45a8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.351+0000 7f45aed25640 1 -- 172.21.15.63:0/340538741 shutdown_connections 2024-06-15T00:09:27.348 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.351+0000 7f45aed25640 1 -- 172.21.15.63:0/340538741 wait complete. 2024-06-15T00:09:27.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.351+0000 7f45aed25640 1 Processor -- start 2024-06-15T00:09:27.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.351+0000 7f45aed25640 1 -- start start 2024-06-15T00:09:27.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45aed25640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:27.350 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45aed25640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8115610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45aed25640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a8115c10 con 0x7f45a8064d40 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45aed25640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45a810f0a0 con 0x7f45a8065650 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:52382/0 (socket says 172.21.15.63:52382) 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 -- 172.21.15.63:0/1311150475 learned_addr learned my addr 172.21.15.63:0/1311150475 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45a7fff640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8115610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 -- 172.21.15.63:0/1311150475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 msgr2=0x7f45a8115610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8115610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 -- 172.21.15.63:0/1311150475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c005040 con 0x7f45a8064d40 2024-06-15T00:09:27.351 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45a7fff640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8115610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45acac1640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f459c02a600 tx=0x7f459c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f459c038070 con 0x7f45a8064d40 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.352+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45a810f320 con 0x7f45a8064d40 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.353+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f459c009240 con 0x7f45a8064d40 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.353+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45a810f7e0 con 0x7f45a8064d40 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.353+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f459c032c20 con 0x7f45a8064d40 2024-06-15T00:09:27.352 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.353+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45a80fb840 con 0x7f45a8064d40 2024-06-15T00:09:27.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.354+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f459c030020 con 0x7f45a8064d40 2024-06-15T00:09:27.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.354+0000 7f45a57fa640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 0x7f45980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:27.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.355+0000 7f45a7fff640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 0x7f45980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:27.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.355+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f459c0a2120 con 0x7f45a8064d40 2024-06-15T00:09:27.353 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.355+0000 7f45a7fff640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 0x7f45980678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45a8110990 tx=0x7f4594009ed0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:27.355 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.359+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f459c070650 con 0x7f45a8064d40 2024-06-15T00:09:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.567+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f45a804b480 con 0x7f45a8064d40 2024-06-15T00:09:27.565 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.568+0000 7f45a57fa640 1 -- 172.21.15.63:0/1311150475 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f459c073c70 con 0x7f45a8064d40 2024-06-15T00:09:27.565 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:27.565 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 msgr2=0x7f45980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 0x7f45980678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f45a8110990 tx=0x7f4594009ed0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 msgr2=0x7f45a81150d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f459c02a600 tx=0x7f459c009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 shutdown_connections 2024-06-15T00:09:27.567 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4598065430 0x7f45980678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45a8065650 0x7f45a8115610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 --2- 172.21.15.63:0/1311150475 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f45a8064d40 0x7f45a81150d0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 >> 172.21.15.63:0/1311150475 conn(0x7f45a8072110 msgr2=0x7f45a80f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 shutdown_connections 2024-06-15T00:09:27.568 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:27.570+0000 7f45aed25640 1 -- 172.21.15.63:0/1311150475 wait complete. 2024-06-15T00:09:28.129 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:28 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1311150475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:28.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:28 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1311150475' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:29.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:29 smithi136 ceph-mon[35883]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:29.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:29 smithi063 ceph-mon[28882]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:31.157 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:31.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:31 smithi136 ceph-mon[35883]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:31.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:31 smithi063 ceph-mon[28882]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:31.432 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:32.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:32.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:33.118 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:33 smithi063 ceph-mon[28882]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:33.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 -- 172.21.15.63:0/4107601323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c0102c90 msgr2=0x7f12c00fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:33.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 --2- 172.21.15.63:0/4107601323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c0102c90 0x7f12c00fc320 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f12b00048d0 tx=0x7f12b0009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 -- 172.21.15.63:0/4107601323 shutdown_connections 2024-06-15T00:09:33.152 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 --2- 172.21.15.63:0/4107601323 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c0102c90 0x7f12c00fc320 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 --2- 172.21.15.63:0/4107601323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c01022f0 0x7f12c01026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 -- 172.21.15.63:0/4107601323 >> 172.21.15.63:0/4107601323 conn(0x7f12c00f7f00 msgr2=0x7f12c00fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 -- 172.21.15.63:0/4107601323 shutdown_connections 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.120+0000 7f12c541a640 1 -- 172.21.15.63:0/4107601323 wait complete. 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.121+0000 7f12c541a640 1 Processor -- start 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.121+0000 7f12c541a640 1 -- start start 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0102c90 0x7f12c0113440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c0113a40 con 0x7f12c01022f0 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12c010ce80 con 0x7f12c0102c90 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:33.153 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:45202/0 (socket says 172.21.15.63:45202) 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 -- 172.21.15.63:0/1723358223 learned_addr learned my addr 172.21.15.63:0/1723358223 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12be7fc640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0102c90 0x7f12c0113440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 -- 172.21.15.63:0/1723358223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0102c90 msgr2=0x7f12c0113440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0102c90 0x7f12c0113440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 -- 172.21.15.63:0/1723358223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b0005040 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12beffd640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f12a800c830 tx=0x7f12a8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a800a7d0 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12a800ade0 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c010d160 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f12a80113a0 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.122+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c010d700 con 0x7f12c01022f0 2024-06-15T00:09:33.154 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.124+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c00fda60 con 0x7f12c01022f0 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.124+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f12a8012400 con 0x7f12c01022f0 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.124+0000 7f12b77fe640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 0x7f12ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.125+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12a8082620 con 0x7f12c01022f0 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.125+0000 7f12be7fc640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 0x7f12ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.125+0000 7f12be7fc640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 0x7f12ac0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007ac0 tx=0x7f12b0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:33.155 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.129+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f12a8050b50 con 0x7f12c01022f0 2024-06-15T00:09:33.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:33 smithi136 ceph-mon[35883]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:33.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.338+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f12c00fbb00 con 0x7f12c01022f0 2024-06-15T00:09:33.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.338+0000 7f12b77fe640 1 -- 172.21.15.63:0/1723358223 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f12a8054170 con 0x7f12c01022f0 2024-06-15T00:09:33.335 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:33.335 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 msgr2=0x7f12ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 0x7f12ac0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f12b0007ac0 tx=0x7f12b0035040 comp rx=0 tx=0).stop 2024-06-15T00:09:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 msgr2=0x7f12c0112f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f12a800c830 tx=0x7f12a8004430 comp rx=0 tx=0).stop 2024-06-15T00:09:33.337 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 shutdown_connections 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f12ac065430 0x7f12ac0678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12c0102c90 0x7f12c0113440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 --2- 172.21.15.63:0/1723358223 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f12c01022f0 0x7f12c0112f00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 >> 172.21.15.63:0/1723358223 conn(0x7f12c00f7f00 msgr2=0x7f12c00f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 shutdown_connections 2024-06-15T00:09:33.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:33.340+0000 7f12c541a640 1 -- 172.21.15.63:0/1723358223 wait complete. 2024-06-15T00:09:34.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:34 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1723358223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:34.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:34 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1723358223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:35.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:35 smithi136 ceph-mon[35883]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:35.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:35 smithi063 ceph-mon[28882]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:37.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:37.317 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:37.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:37 smithi136 ceph-mon[35883]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:37.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:37 smithi063 ceph-mon[28882]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:38.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- 172.21.15.63:0/1915613005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fbe80 msgr2=0x7f6bd40fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:38.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/1915613005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fbe80 0x7f6bd40fcd80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6bc80048d0 tx=0x7f6bc8009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:38.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- 172.21.15.63:0/1915613005 shutdown_connections 2024-06-15T00:09:38.953 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/1915613005 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fbe80 0x7f6bd40fcd80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:38.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/1915613005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fb570 0x7f6bd40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:38.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- 172.21.15.63:0/1915613005 >> 172.21.15.63:0/1915613005 conn(0x7f6bd4072110 msgr2=0x7f6bd4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:38.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- 172.21.15.63:0/1915613005 shutdown_connections 2024-06-15T00:09:38.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- 172.21.15.63:0/1915613005 wait complete. 2024-06-15T00:09:38.954 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 Processor -- start 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.952+0000 7f6bdc75b640 1 -- start start 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fbe80 0x7f6bd4113490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd4113a90 con 0x7f6bd40fb570 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd410ced0 con 0x7f6bd40fbe80 2024-06-15T00:09:38.955 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:38.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49372/0 (socket says 172.21.15.63:49372) 2024-06-15T00:09:38.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 -- 172.21.15.63:0/666555018 learned_addr learned my addr 172.21.15.63:0/666555018 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:38.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bd9cf6640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fbe80 0x7f6bd4113490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:38.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 -- 172.21.15.63:0/666555018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fbe80 msgr2=0x7f6bd4113490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:38.956 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fbe80 0x7f6bd4113490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 -- 172.21.15.63:0/666555018 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bc8005040 con 0x7f6bd40fb570 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bda4f7640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400db00 tx=0x7f6bc4004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc4017070 con 0x7f6bd40fb570 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bc400f040 con 0x7f6bd40fb570 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd410d1b0 con 0x7f6bd40fb570 2024-06-15T00:09:38.957 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bc40039b0 con 0x7f6bd40fb570 2024-06-15T00:09:38.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.953+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd410d750 con 0x7f6bd40fb570 2024-06-15T00:09:38.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.955+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd40fc920 con 0x7f6bd40fb570 2024-06-15T00:09:38.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.955+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f6bc4003ba0 con 0x7f6bd40fb570 2024-06-15T00:09:38.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.956+0000 7f6bd2ffd640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:38.958 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.956+0000 7f6bd9cf6640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:38.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.956+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6bc4082550 con 0x7f6bd40fb570 2024-06-15T00:09:38.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.956+0000 7f6bd9cf6640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 0x7f6bbc0678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8004540 tx=0x7f6bc8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:38.959 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:38.960+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6bc4050a80 con 0x7f6bd40fb570 2024-06-15T00:09:39.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.173+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6bd4072c20 con 0x7f6bd40fb570 2024-06-15T00:09:39.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.174+0000 7f6bd2ffd640 1 -- 172.21.15.63:0/666555018 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6bc40540a0 con 0x7f6bd40fb570 2024-06-15T00:09:39.171 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:39.171 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 msgr2=0x7f6bbc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 0x7f6bbc0678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8004540 tx=0x7f6bc8035040 comp rx=0 tx=0).stop 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 msgr2=0x7f6bd4112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6bc400db00 tx=0x7f6bc4004e60 comp rx=0 tx=0).stop 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 shutdown_connections 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f6bbc065430 0x7f6bbc0678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6bd40fbe80 0x7f6bd4113490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 --2- 172.21.15.63:0/666555018 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f6bd40fb570 0x7f6bd4112f50 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:39.173 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 >> 172.21.15.63:0/666555018 conn(0x7f6bd4072110 msgr2=0x7f6bd4065570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 shutdown_connections 2024-06-15T00:09:39.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:39.176+0000 7f6bdc75b640 1 -- 172.21.15.63:0/666555018 wait complete. 2024-06-15T00:09:39.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:39 smithi136 ceph-mon[35883]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:39.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:39 smithi063 ceph-mon[28882]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:40.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:40 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/666555018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:40.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:40 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/666555018' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:41.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:41 smithi063 ceph-mon[28882]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:41.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:41 smithi136 ceph-mon[35883]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:42.870 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:43.147 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:43.407 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:43 smithi063 ceph-mon[28882]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:43.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:43 smithi136 ceph-mon[35883]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 -- 172.21.15.63:0/1516841461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c102c90 msgr2=0x7f513c0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 --2- 172.21.15.63:0/1516841461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c102c90 0x7f513c0fc320 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f51240048d0 tx=0x7f5124009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:44.810 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 -- 172.21.15.63:0/1516841461 shutdown_connections 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 --2- 172.21.15.63:0/1516841461 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c102c90 0x7f513c0fc320 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 --2- 172.21.15.63:0/1516841461 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c1022f0 0x7f513c1026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 -- 172.21.15.63:0/1516841461 >> 172.21.15.63:0/1516841461 conn(0x7f513c0f7f00 msgr2=0x7f513c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 -- 172.21.15.63:0/1516841461 shutdown_connections 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.808+0000 7f5140b26640 1 -- 172.21.15.63:0/1516841461 wait complete. 2024-06-15T00:09:44.811 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 Processor -- start 2024-06-15T00:09:44.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 -- start start 2024-06-15T00:09:44.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:44.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c102c90 0x7f513c1134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:44.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c113b50 con 0x7f513c1022f0 2024-06-15T00:09:44.812 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.809+0000 7f5140b26640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c10cf00 con 0x7f513c102c90 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:49400/0 (socket says 172.21.15.63:49400) 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 -- 172.21.15.63:0/1268892170 learned_addr learned my addr 172.21.15.63:0/1268892170 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5133fff640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c102c90 0x7f513c1134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 -- 172.21.15.63:0/1268892170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c102c90 msgr2=0x7f513c1134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:44.813 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c102c90 0x7f513c1134c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 -- 172.21.15.63:0/1268892170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5124005040 con 0x7f513c1022f0 2024-06-15T00:09:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f513a59c640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b790 tx=0x7f512c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512c00c700 con 0x7f513c1022f0 2024-06-15T00:09:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f512c00cd10 con 0x7f513c1022f0 2024-06-15T00:09:44.814 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f513c10d1e0 con 0x7f513c1022f0 2024-06-15T00:09:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f512c013370 con 0x7f513c1022f0 2024-06-15T00:09:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.810+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513c10d700 con 0x7f513c1022f0 2024-06-15T00:09:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.811+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f513c0fda60 con 0x7f513c1022f0 2024-06-15T00:09:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.812+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f512c012070 con 0x7f513c1022f0 2024-06-15T00:09:44.815 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.812+0000 7f5132ffd640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 0x7f51280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:44.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.812+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f512c081e70 con 0x7f513c1022f0 2024-06-15T00:09:44.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.812+0000 7f5133fff640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 0x7f51280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:44.816 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.813+0000 7f5133fff640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 0x7f51280678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f512400b020 tx=0x7f5124035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:44.817 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:44.816+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f512c0503a0 con 0x7f513c1022f0 2024-06-15T00:09:45.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.024+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f513c10d370 con 0x7f513c1022f0 2024-06-15T00:09:45.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.025+0000 7f5132ffd640 1 -- 172.21.15.63:0/1268892170 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f513c10d370 con 0x7f513c1022f0 2024-06-15T00:09:45.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:45.022 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 msgr2=0x7f51280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 0x7f51280678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f512400b020 tx=0x7f5124035040 comp rx=0 tx=0).stop 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 msgr2=0x7f513c112f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f512c00b790 tx=0x7f512c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 shutdown_connections 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5128065430 0x7f51280678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f513c102c90 0x7f513c1134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 --2- 172.21.15.63:0/1268892170 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f513c1022f0 0x7f513c112f80 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 >> 172.21.15.63:0/1268892170 conn(0x7f513c0f7f00 msgr2=0x7f513c0f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:45.024 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 shutdown_connections 2024-06-15T00:09:45.025 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:45.027+0000 7f5140b26640 1 -- 172.21.15.63:0/1268892170 wait complete. 2024-06-15T00:09:45.319 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:45 smithi063 ceph-mon[28882]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:45.319 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:45 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1268892170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:45.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:45 smithi136 ceph-mon[35883]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:45.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:45 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1268892170' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:47.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:47 smithi063 ceph-mon[28882]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:47.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:47 smithi136 ceph-mon[35883]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:09:48.740 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:49.022 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:49.397 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:49 smithi063 ceph-mon[28882]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:49.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:49 smithi136 ceph-mon[35883]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:50.731 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 -- 172.21.15.63:0/610669496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 msgr2=0x7f96140fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 --2- 172.21.15.63:0/610669496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fa9e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f96100048d0 tx=0x7f9610009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 -- 172.21.15.63:0/610669496 shutdown_connections 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 --2- 172.21.15.63:0/610669496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fa9e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 --2- 172.21.15.63:0/610669496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96141026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 -- 172.21.15.63:0/610669496 >> 172.21.15.63:0/610669496 conn(0x7f9614070a40 msgr2=0x7f9614070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 -- 172.21.15.63:0/610669496 shutdown_connections 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 -- 172.21.15.63:0/610669496 wait complete. 2024-06-15T00:09:50.732 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.607+0000 7f961c586640 1 Processor -- start 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961c586640 1 -- start start 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961c586640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96140fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961c586640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961c586640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140fc640 con 0x7f9614102c90 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961c586640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140fc7b0 con 0x7f96141022f0 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39678/0 (socket says 172.21.15.63:39678) 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 -- 172.21.15.63:0/3653479666 learned_addr learned my addr 172.21.15.63:0/3653479666 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:50.733 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961a322640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96140fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 -- 172.21.15.63:0/3653479666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 msgr2=0x7f96140fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96140fbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 -- 172.21.15.63:0/3653479666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610005040 con 0x7f9614102c90 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f961a322640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96140fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9619b21640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f96100043d0 tx=0x7f9610009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.608+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9610038070 con 0x7f9614102c90 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.609+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f961002cf00 con 0x7f9614102c90 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.609+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96140fc9e0 con 0x7f9614102c90 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.609+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9610009480 con 0x7f9614102c90 2024-06-15T00:09:50.734 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.609+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141195e0 con 0x7f9614102c90 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.610+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9614065510 con 0x7f9614102c90 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.610+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f9610030020 con 0x7f9614102c90 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.611+0000 7f9606ffd640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.611+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f96100a21c0 con 0x7f9614102c90 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.611+0000 7f961a322640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.611+0000 7f961a322640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 0x7f96000678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9608007fd0 tx=0x7f9608009e70 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:50.735 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.615+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f96100706f0 con 0x7f9614102c90 2024-06-15T00:09:50.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.824+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9614074960 con 0x7f9614102c90 2024-06-15T00:09:50.821 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.824+0000 7f9606ffd640 1 -- 172.21.15.63:0/3653479666 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9610073d10 con 0x7f9614102c90 2024-06-15T00:09:50.821 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:50.821 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 msgr2=0x7f96000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 0x7f96000678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9608007fd0 tx=0x7f9608009e70 comp rx=0 tx=0).stop 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 msgr2=0x7f96140fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f96100043d0 tx=0x7f9610009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 shutdown_connections 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.823 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f9614102c90 0x7f96140fc100 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 --2- 172.21.15.63:0/3653479666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f96141022f0 0x7f96140fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:50.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 >> 172.21.15.63:0/3653479666 conn(0x7f9614070a40 msgr2=0x7f96140f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:50.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 shutdown_connections 2024-06-15T00:09:50.824 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:50.826+0000 7f961c586640 1 -- 172.21.15.63:0/3653479666 wait complete. 2024-06-15T00:09:51.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:51 smithi063 ceph-mon[28882]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:51.376 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:51 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3653479666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:51.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:51 smithi136 ceph-mon[35883]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:51.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:51 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3653479666' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:53.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:53 smithi063 ceph-mon[28882]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:53.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:53 smithi136 ceph-mon[35883]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:54.404 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:09:54.683 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:09:55.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:55 smithi063 ceph-mon[28882]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:55.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:55 smithi136 ceph-mon[35883]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 -- 172.21.15.63:0/2906192075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a41022f0 msgr2=0x7f18a41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/2906192075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a41022f0 0x7f18a41026c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f18940048d0 tx=0x7f1894009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 -- 172.21.15.63:0/2906192075 shutdown_connections 2024-06-15T00:09:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/2906192075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a4102c90 0x7f18a40fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.312 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/2906192075 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a41022f0 0x7f18a41026c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.295+0000 7f18a8bdd640 1 -- 172.21.15.63:0/2906192075 >> 172.21.15.63:0/2906192075 conn(0x7f18a4070a40 msgr2=0x7f18a4070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 -- 172.21.15.63:0/2906192075 shutdown_connections 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 -- 172.21.15.63:0/2906192075 wait complete. 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 Processor -- start 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 -- start start 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a41022f0 0x7f18a4113050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a4113c20 con 0x7f18a4102c90 2024-06-15T00:09:56.313 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f18a8bdd640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a410cfd0 con 0x7f18a41022f0 2024-06-15T00:09:56.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f189bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:56.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f189bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:39698/0 (socket says 172.21.15.63:39698) 2024-06-15T00:09:56.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.296+0000 7f189bfff640 1 -- 172.21.15.63:0/4035913404 learned_addr learned my addr 172.21.15.63:0/4035913404 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:09:56.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f18a259c640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a41022f0 0x7f18a4113050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:56.314 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189bfff640 1 -- 172.21.15.63:0/4035913404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a41022f0 msgr2=0x7f18a4113050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189bfff640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a41022f0 0x7f18a4113050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189bfff640 1 -- 172.21.15.63:0/4035913404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1894005040 con 0x7f18a4102c90 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189bfff640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f188c00b540 tx=0x7f188c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f188c00f6f0 con 0x7f18a4102c90 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f188c00fd00 con 0x7f18a4102c90 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a410d2b0 con 0x7f18a4102c90 2024-06-15T00:09:56.315 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f188c0103a0 con 0x7f18a4102c90 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.297+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a410d7d0 con 0x7f18a4102c90 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.298+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a4065510 con 0x7f18a4102c90 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.298+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f188c00f850 con 0x7f18a4102c90 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.299+0000 7f189affd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 0x7f18900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.299+0000 7f18a259c640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 0x7f18900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.299+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f188c0823a0 con 0x7f18a4102c90 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.300+0000 7f18a259c640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 0x7f18900678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1894007ac0 tx=0x7f1894009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:09:56.316 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.303+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f188c050970 con 0x7f18a4102c90 2024-06-15T00:09:56.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.511+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f18a4109410 con 0x7f18a4102c90 2024-06-15T00:09:56.508 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.511+0000 7f189affd640 1 -- 172.21.15.63:0/4035913404 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f188c053f90 con 0x7f18a4102c90 2024-06-15T00:09:56.509 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:09:56.509 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 msgr2=0x7f18900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 0x7f18900678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1894007ac0 tx=0x7f1894009da0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 msgr2=0x7f18a4113590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f188c00b540 tx=0x7f188c0091a0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 shutdown_connections 2024-06-15T00:09:56.510 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1890065430 0x7f18900678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f18a4102c90 0x7f18a4113590 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 --2- 172.21.15.63:0/4035913404 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18a41022f0 0x7f18a4113050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:09:56.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 >> 172.21.15.63:0/4035913404 conn(0x7f18a4070a40 msgr2=0x7f18a40f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:09:56.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 shutdown_connections 2024-06-15T00:09:56.511 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:09:56.513+0000 7f18a8bdd640 1 -- 172.21.15.63:0/4035913404 wait complete. 2024-06-15T00:09:57.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:57 smithi063 ceph-mon[28882]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:57.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:57 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/4035913404' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:57.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:57 smithi136 ceph-mon[35883]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:57.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:57 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/4035913404' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:09:59.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:09:59 smithi063 ceph-mon[28882]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:09:59.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:09:59 smithi136 ceph-mon[35883]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:00.083 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:00.364 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:00.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:00 smithi063 ceph-mon[28882]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:10:00.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:00 smithi136 ceph-mon[35883]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:10:01.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:01 smithi063 ceph-mon[28882]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:01.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:01 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:10:01.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:01 smithi136 ceph-mon[35883]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:01.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:01 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1856283868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fa9e0 msgr2=0x7fcf340fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1856283868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fa9e0 0x7fcf340fae30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcf240048d0 tx=0x7fcf24009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1856283868 shutdown_connections 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1856283868 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fa9e0 0x7fcf340fae30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1856283868 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fd100 0x7fcf340fd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1856283868 >> 172.21.15.63:0/1856283868 conn(0x7fcf34070a40 msgr2=0x7fcf34070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1856283868 shutdown_connections 2024-06-15T00:10:02.098 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.099+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1856283868 wait complete. 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 Processor -- start 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 -- start start 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 0x7fcf34115180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf34115cc0 con 0x7fcf340fd100 2024-06-15T00:10:02.099 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf3bb30640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf3410f100 con 0x7fcf340fa9e0 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59112/0 (socket says 172.21.15.63:59112) 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 -- 172.21.15.63:0/1881937147 learned_addr learned my addr 172.21.15.63:0/1881937147 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf398cc640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 0x7fcf34115180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 -- 172.21.15.63:0/1881937147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 msgr2=0x7fcf34115180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:02.101 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 0x7fcf34115180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.100+0000 7fcf390cb640 1 -- 172.21.15.63:0/1881937147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf24005040 con 0x7fcf340fd100 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf398cc640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 0x7fcf34115180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf390cb640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcf240048a0 tx=0x7fcf24009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf24038070 con 0x7fcf340fd100 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf2402cef0 con 0x7fcf340fd100 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf3410f380 con 0x7fcf340fd100 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcf24009640 con 0x7fcf340fd100 2024-06-15T00:10:02.102 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.101+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf3410f840 con 0x7fcf340fd100 2024-06-15T00:10:02.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.102+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf340fb9f0 con 0x7fcf340fd100 2024-06-15T00:10:02.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.103+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fcf24030020 con 0x7fcf340fd100 2024-06-15T00:10:02.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.103+0000 7fcf2a7fc640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 0x7fcf200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:02.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.103+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcf240a2890 con 0x7fcf340fd100 2024-06-15T00:10:02.103 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.103+0000 7fcf398cc640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 0x7fcf200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:02.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.104+0000 7fcf398cc640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 0x7fcf200678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf30007fd0 tx=0x7fcf30009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:02.104 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.108+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcf24070dc0 con 0x7fcf340fd100 2024-06-15T00:10:02.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.324+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcf3410b500 con 0x7fcf340fd100 2024-06-15T00:10:02.321 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.324+0000 7fcf2a7fc640 1 -- 172.21.15.63:0/1881937147 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcf240743e0 con 0x7fcf340fd100 2024-06-15T00:10:02.322 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:02.322 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:02.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 msgr2=0x7fcf200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:02.323 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 0x7fcf200678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fcf30007fd0 tx=0x7fcf30009e90 comp rx=0 tx=0).stop 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 msgr2=0x7fcf341156c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcf240048a0 tx=0x7fcf24009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 shutdown_connections 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fcf20065430 0x7fcf200678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fcf340fd100 0x7fcf341156c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 --2- 172.21.15.63:0/1881937147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf340fa9e0 0x7fcf34115180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 >> 172.21.15.63:0/1881937147 conn(0x7fcf34070a40 msgr2=0x7fcf340f89d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 shutdown_connections 2024-06-15T00:10:02.324 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:02.326+0000 7fcf3bb30640 1 -- 172.21.15.63:0/1881937147 wait complete. 2024-06-15T00:10:02.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:10:02.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:10:02.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:10:02.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:10:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:10:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:10:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:03.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:03 smithi063 ceph-mon[28882]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:03.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:03 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1881937147' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:03.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:03 smithi136 ceph-mon[35883]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:03.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:03 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1881937147' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:05.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:05 smithi063 ceph-mon[28882]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:05.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:05 smithi136 ceph-mon[35883]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:05.997 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:06.276 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:07.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:07 smithi063 ceph-mon[28882]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:07.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:07 smithi136 ceph-mon[35883]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 -- 172.21.15.63:0/777576680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f41023e0 msgr2=0x7fc8f41027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:07.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 --2- 172.21.15.63:0/777576680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f41023e0 0x7fc8f41027b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40048d0 tx=0x7fc8e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 -- 172.21.15.63:0/777576680 shutdown_connections 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 --2- 172.21.15.63:0/777576680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f40faa50 0x7fc8f40faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 --2- 172.21.15.63:0/777576680 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f41023e0 0x7fc8f41027b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 -- 172.21.15.63:0/777576680 >> 172.21.15.63:0/777576680 conn(0x7fc8f4070ab0 msgr2=0x7fc8f4070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 -- 172.21.15.63:0/777576680 shutdown_connections 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.746+0000 7fc8fa944640 1 -- 172.21.15.63:0/777576680 wait complete. 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 Processor -- start 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 -- start start 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f41023e0 0x7fc8f41135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f4113c50 con 0x7fc8f40faa50 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8fa944640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f410d000 con 0x7fc8f41023e0 2024-06-15T00:10:07.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:59128/0 (socket says 172.21.15.63:59128) 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 -- 172.21.15.63:0/669013324 learned_addr learned my addr 172.21.15.63:0/669013324 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f37fe640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f41023e0 0x7fc8f41135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 -- 172.21.15.63:0/669013324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f41023e0 msgr2=0x7fc8f41135c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f41023e0 0x7fc8f41135c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.747+0000 7fc8f3fff640 1 -- 172.21.15.63:0/669013324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e4005040 con 0x7fc8f40faa50 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8f3fff640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40048a0 tx=0x7fc8e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e4038070 con 0x7fc8f40faa50 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e4032d30 con 0x7fc8f40faa50 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f410d280 con 0x7fc8f40faa50 2024-06-15T00:10:07.769 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e4009680 con 0x7fc8f40faa50 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.748+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f410d740 con 0x7fc8f40faa50 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.749+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f4065550 con 0x7fc8f40faa50 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.749+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc8e4030020 con 0x7fc8f40faa50 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.750+0000 7fc8f0ff9640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.750+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc8e40a2260 con 0x7fc8f40faa50 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.750+0000 7fc8f37fe640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.750+0000 7fc8f37fe640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 0x7fc8dc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006fd0 tx=0x7fc8e8008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:07.770 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.754+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc8e4070710 con 0x7fc8f40faa50 2024-06-15T00:10:07.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.967+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc8f40749d0 con 0x7fc8f40faa50 2024-06-15T00:10:07.964 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.967+0000 7fc8f0ff9640 1 -- 172.21.15.63:0/669013324 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc8e4073d30 con 0x7fc8f40faa50 2024-06-15T00:10:07.964 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:07.965 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 msgr2=0x7fc8dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 0x7fc8dc0678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc8e8006fd0 tx=0x7fc8e8008040 comp rx=0 tx=0).stop 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 msgr2=0x7fc8f4113080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc8e40048a0 tx=0x7fc8e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 shutdown_connections 2024-06-15T00:10:07.967 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8f41023e0 0x7fc8f41135c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 --2- 172.21.15.63:0/669013324 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc8f40faa50 0x7fc8f4113080 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:07.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 >> 172.21.15.63:0/669013324 conn(0x7fc8f4070ab0 msgr2=0x7fc8f40f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:07.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.969+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 shutdown_connections 2024-06-15T00:10:07.968 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:07.970+0000 7fc8fa944640 1 -- 172.21.15.63:0/669013324 wait complete. 2024-06-15T00:10:08.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:08 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/669013324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:08.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:08 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/669013324' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:09.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:09 smithi063 ceph-mon[28882]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:09.485 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:09 smithi136 ceph-mon[35883]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:11.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:11 smithi063 ceph-mon[28882]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:11.541 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:11.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:11 smithi136 ceph-mon[35883]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:11.821 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:13.178 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:13 smithi063 ceph-mon[28882]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.411+0000 7f2ad5b93640 1 -- 172.21.15.63:0/4151852312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad0065030 msgr2=0x7f2ad0065400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.411+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/4151852312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad0065030 0x7f2ad0065400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2ac00048d0 tx=0x7f2ac0009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.411+0000 7f2ad5b93640 1 -- 172.21.15.63:0/4151852312 shutdown_connections 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/4151852312 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fe310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/4151852312 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad0065030 0x7f2ad0065400 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 -- 172.21.15.63:0/4151852312 >> 172.21.15.63:0/4151852312 conn(0x7f2ad0072110 msgr2=0x7f2ad0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 -- 172.21.15.63:0/4151852312 shutdown_connections 2024-06-15T00:10:13.409 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 -- 172.21.15.63:0/4151852312 wait complete. 2024-06-15T00:10:13.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 Processor -- start 2024-06-15T00:10:13.410 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.412+0000 7f2ad5b93640 1 -- start start 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad0065030 0x7f2ad00fb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00fbd40 con 0x7f2ad00fdea0 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad00fbeb0 con 0x7f2ad0065030 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:54060/0 (socket says 172.21.15.63:54060) 2024-06-15T00:10:13.411 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 -- 172.21.15.63:0/3443049971 learned_addr learned my addr 172.21.15.63:0/3443049971 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2acf7fe640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad0065030 0x7f2ad00fb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 -- 172.21.15.63:0/3443049971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad0065030 msgr2=0x7f2ad00fb2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad0065030 0x7f2ad00fb2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 -- 172.21.15.63:0/3443049971 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ac0005040 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2aceffd640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400c790 tx=0x7f2ac40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ac400a7a0 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad01192f0 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ac400adb0 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ac40113a0 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.413+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad0119670 con 0x7f2ad00fdea0 2024-06-15T00:10:13.412 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.414+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad00fda40 con 0x7f2ad00fdea0 2024-06-15T00:10:13.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.415+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2ac400a900 con 0x7f2ad00fdea0 2024-06-15T00:10:13.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.415+0000 7f2ab7fff640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 0x7f2ab80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:13.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.415+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2ac4082550 con 0x7f2ad00fdea0 2024-06-15T00:10:13.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.415+0000 7f2acf7fe640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 0x7f2ab80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:13.413 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.416+0000 7f2acf7fe640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 0x7f2ab80678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:13.416 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.420+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2ac4050a80 con 0x7f2ad00fdea0 2024-06-15T00:10:13.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:13 smithi136 ceph-mon[35883]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:13.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.628+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2ad0072c20 con 0x7f2ad00fdea0 2024-06-15T00:10:13.625 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.628+0000 7f2ab7fff640 1 -- 172.21.15.63:0/3443049971 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2ac40540a0 con 0x7f2ad00fdea0 2024-06-15T00:10:13.626 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:13.626 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.630+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 msgr2=0x7f2ab80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.630+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 0x7f2ab80678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f2ac0000c00 tx=0x7f2ac0035040 comp rx=0 tx=0).stop 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.630+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 msgr2=0x7f2ad00fb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.630+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2ac400c790 tx=0x7f2ac40043c0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.630+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 shutdown_connections 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2ab8065430 0x7f2ab80678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2ad00fdea0 0x7f2ad00fb800 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 --2- 172.21.15.63:0/3443049971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ad0065030 0x7f2ad00fb2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 >> 172.21.15.63:0/3443049971 conn(0x7f2ad0072110 msgr2=0x7f2ad0105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 shutdown_connections 2024-06-15T00:10:13.628 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:13.631+0000 7f2ad5b93640 1 -- 172.21.15.63:0/3443049971 wait complete. 2024-06-15T00:10:14.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:14 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3443049971' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:14.432 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:14 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3443049971' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:15.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:15 smithi063 ceph-mon[28882]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:15.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:15 smithi136 ceph-mon[35883]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:17.233 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:17 smithi063 ceph-mon[28882]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:17.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:17.512 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:17 smithi136 ceph-mon[35883]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:19.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.016+0000 7f56eb9d6640 1 -- 172.21.15.63:0/723241201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 msgr2=0x7f56e40fb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:19.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.016+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/723241201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e40fb420 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f56e00048d0 tx=0x7f56e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 -- 172.21.15.63:0/723241201 shutdown_connections 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/723241201 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 0x7f56e41086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/723241201 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e40fb420 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 -- 172.21.15.63:0/723241201 >> 172.21.15.63:0/723241201 conn(0x7f56e40f6e00 msgr2=0x7f56e40f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 -- 172.21.15.63:0/723241201 shutdown_connections 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 -- 172.21.15.63:0/723241201 wait complete. 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 Processor -- start 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 -- start start 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.017+0000 7f56eb9d6640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56eb9d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 0x7f56e4115630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:19.018 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56eb9d6640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4115c30 con 0x7f56e40fb050 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56eb9d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e410f070 con 0x7f56e40fb960 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38458/0 (socket says 172.21.15.63:38458) 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 -- 172.21.15.63:0/1945362259 learned_addr learned my addr 172.21.15.63:0/1945362259 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e8f71640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 0x7f56e4115630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 -- 172.21.15.63:0/1945362259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 msgr2=0x7f56e4115630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 0x7f56e4115630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 -- 172.21.15.63:0/1945362259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e0005040 con 0x7f56e40fb050 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56e9772640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56e002ca80 tx=0x7f56e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e0038070 con 0x7f56e40fb050 2024-06-15T00:10:19.019 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e410f2f0 con 0x7f56e40fb050 2024-06-15T00:10:19.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f56e002ae90 con 0x7f56e40fb050 2024-06-15T00:10:19.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.018+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f56e0032be0 con 0x7f56e40fb050 2024-06-15T00:10:19.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.019+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e410f830 con 0x7f56e40fb050 2024-06-15T00:10:19.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.020+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e40660b0 con 0x7f56e40fb050 2024-06-15T00:10:19.020 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.020+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f56e0030020 con 0x7f56e40fb050 2024-06-15T00:10:19.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.021+0000 7f56d9ffb640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 0x7f56d00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:19.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.021+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f56e00a2260 con 0x7f56e40fb050 2024-06-15T00:10:19.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.021+0000 7f56e8f71640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 0x7f56d00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:19.021 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.022+0000 7f56e8f71640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 0x7f56d00678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f56e41109b0 tx=0x7f56d4009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:19.022 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.025+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f56e00707c0 con 0x7f56e40fb050 2024-06-15T00:10:19.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.235+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f56e404b480 con 0x7f56e40fb050 2024-06-15T00:10:19.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.235+0000 7f56d9ffb640 1 -- 172.21.15.63:0/1945362259 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f56e0073de0 con 0x7f56e40fb050 2024-06-15T00:10:19.232 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:19.233 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:19.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 msgr2=0x7f56d00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:19.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 0x7f56d00678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f56e41109b0 tx=0x7f56d4009040 comp rx=0 tx=0).stop 2024-06-15T00:10:19.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 msgr2=0x7f56e41150f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f56e002ca80 tx=0x7f56e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 shutdown_connections 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f56d0065430 0x7f56d00678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56e40fb960 0x7f56e4115630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 --2- 172.21.15.63:0/1945362259 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f56e40fb050 0x7f56e41150f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.237+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 >> 172.21.15.63:0/1945362259 conn(0x7f56e40f6e00 msgr2=0x7f56e40f7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.238+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 shutdown_connections 2024-06-15T00:10:19.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:19.238+0000 7f56eb9d6640 1 -- 172.21.15.63:0/1945362259 wait complete. 2024-06-15T00:10:19.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:19 smithi063 ceph-mon[28882]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:19.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:19 smithi136 ceph-mon[35883]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:20.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:20 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1945362259' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:20.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:20 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1945362259' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:21.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:21 smithi063 ceph-mon[28882]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:21.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:21 smithi136 ceph-mon[35883]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:22.873 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:23.153 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:23.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:23 smithi063 ceph-mon[28882]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:23.586 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:23 smithi136 ceph-mon[35883]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:24.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 -- 172.21.15.63:0/4183115520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15801022d0 msgr2=0x7f15801026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:24.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 --2- 172.21.15.63:0/4183115520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15801022d0 0x7f15801026a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f15700048d0 tx=0x7f1570009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.630 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 -- 172.21.15.63:0/4183115520 shutdown_connections 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 --2- 172.21.15.63:0/4183115520 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1580102c70 0x7f15800fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 --2- 172.21.15.63:0/4183115520 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f15801022d0 0x7f15801026a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 -- 172.21.15.63:0/4183115520 >> 172.21.15.63:0/4183115520 conn(0x7f15800f7e80 msgr2=0x7f15800fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 -- 172.21.15.63:0/4183115520 shutdown_connections 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.621+0000 7f1586153640 1 -- 172.21.15.63:0/4183115520 wait complete. 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 Processor -- start 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 -- start start 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15801022d0 0x7f1580112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1580113a50 con 0x7f1580102c70 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.622+0000 7f1586153640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f158010ce90 con 0x7f15801022d0 2024-06-15T00:10:24.631 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15801022d0 0x7f1580112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:38486/0 (socket says 172.21.15.63:38486) 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 -- 172.21.15.63:0/2006961181 learned_addr learned my addr 172.21.15.63:0/2006961181 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 -- 172.21.15.63:0/2006961181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15801022d0 msgr2=0x7f1580112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15801022d0 0x7f1580112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 -- 172.21.15.63:0/2006961181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1570005040 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f157effd640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f157400c810 tx=0x7f15740043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f157400a7a0 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f157400adb0 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1574011380 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158010d170 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.623+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158010d710 con 0x7f1580102c70 2024-06-15T00:10:24.632 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.624+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15800fda40 con 0x7f1580102c70 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.625+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f1574012420 con 0x7f1580102c70 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.625+0000 7f1567fff640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 0x7f15680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.625+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1574082950 con 0x7f1580102c70 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.625+0000 7f157f7fe640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 0x7f15680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.626+0000 7f157f7fe640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 0x7f15680678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f157002a600 tx=0x7f1570035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:24.633 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.629+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1574050f40 con 0x7f1580102c70 2024-06-15T00:10:24.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.839+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f158010e580 con 0x7f1580102c70 2024-06-15T00:10:24.836 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.840+0000 7f1567fff640 1 -- 172.21.15.63:0/2006961181 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1574054560 con 0x7f1580102c70 2024-06-15T00:10:24.837 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:24.837 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 msgr2=0x7f15680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 0x7f15680678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f157002a600 tx=0x7f1570035040 comp rx=0 tx=0).stop 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 msgr2=0x7f1580113450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f157400c810 tx=0x7f15740043c0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 shutdown_connections 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f1568065430 0x7f15680678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f1580102c70 0x7f1580113450 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 --2- 172.21.15.63:0/2006961181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15801022d0 0x7f1580112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:24.839 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 >> 172.21.15.63:0/2006961181 conn(0x7f15800f7e80 msgr2=0x7f15800f8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:24.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 shutdown_connections 2024-06-15T00:10:24.840 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:24.842+0000 7f1586153640 1 -- 172.21.15.63:0/2006961181 wait complete. 2024-06-15T00:10:25.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:25 smithi063 ceph-mon[28882]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:25.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:25 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2006961181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:25.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:25 smithi136 ceph-mon[35883]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:25.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:25 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2006961181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:27.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:27 smithi063 ceph-mon[28882]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:27.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:27 smithi136 ceph-mon[35883]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:28.472 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:28.752 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:29.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:29 smithi063 ceph-mon[28882]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:29.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:29 smithi136 ceph-mon[35883]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:30.260 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 -- 172.21.15.63:0/3412314305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc948102c90 msgr2=0x7fc9480fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/3412314305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc948102c90 0x7fc9480fa9e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc9440048d0 tx=0x7fc944009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 -- 172.21.15.63:0/3412314305 shutdown_connections 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/3412314305 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc948102c90 0x7fc9480fa9e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/3412314305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9481022f0 0x7fc9481026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 -- 172.21.15.63:0/3412314305 >> 172.21.15.63:0/3412314305 conn(0x7fc948070a40 msgr2=0x7fc948070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 -- 172.21.15.63:0/3412314305 shutdown_connections 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.251+0000 7fc94fd7b640 1 -- 172.21.15.63:0/3412314305 wait complete. 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 Processor -- start 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 -- start start 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc948102c90 0x7fc948113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc948113b20 con 0x7fc9481022f0 2024-06-15T00:10:30.261 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94fd7b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc94810cf60 con 0x7fc948102c90 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94db17640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94db17640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47794/0 (socket says 172.21.15.63:47794) 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94db17640 1 -- 172.21.15.63:0/653248774 learned_addr learned my addr 172.21.15.63:0/653248774 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.252+0000 7fc94d316640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc948102c90 0x7fc948113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94db17640 1 -- 172.21.15.63:0/653248774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc948102c90 msgr2=0x7fc948113520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94db17640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc948102c90 0x7fc948113520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94db17640 1 -- 172.21.15.63:0/653248774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc944005040 con 0x7fc9481022f0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94db17640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c810 tx=0x7fc9380043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc93800a7a0 con 0x7fc9481022f0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc93800adb0 con 0x7fc9481022f0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc94810d240 con 0x7fc9481022f0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc938011380 con 0x7fc9481022f0 2024-06-15T00:10:30.262 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.253+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc94810d730 con 0x7fc9481022f0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.254+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc948065510 con 0x7fc9481022f0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.254+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fc938012420 con 0x7fc9481022f0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.255+0000 7fc93e7fc640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 0x7fc9340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.255+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc938082700 con 0x7fc9481022f0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.255+0000 7fc94d316640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 0x7fc9340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.255+0000 7fc94d316640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 0x7fc9340678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc948103190 tx=0x7fc944035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:30.263 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.259+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc938050cf0 con 0x7fc9481022f0 2024-06-15T00:10:30.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.469+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc948109340 con 0x7fc9481022f0 2024-06-15T00:10:30.466 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.470+0000 7fc93e7fc640 1 -- 172.21.15.63:0/653248774 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc938054310 con 0x7fc9481022f0 2024-06-15T00:10:30.467 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:30.467 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 msgr2=0x7fc9340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 0x7fc9340678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc948103190 tx=0x7fc944035040 comp rx=0 tx=0).stop 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 msgr2=0x7fc948112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc93800c810 tx=0x7fc9380043c0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 shutdown_connections 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fc934065430 0x7fc9340678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc948102c90 0x7fc948113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 --2- 172.21.15.63:0/653248774 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fc9481022f0 0x7fc948112fe0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 >> 172.21.15.63:0/653248774 conn(0x7fc948070a40 msgr2=0x7fc9480f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 shutdown_connections 2024-06-15T00:10:30.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:30.472+0000 7fc94fd7b640 1 -- 172.21.15.63:0/653248774 wait complete. 2024-06-15T00:10:31.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:31 smithi063 ceph-mon[28882]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:31.428 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:31 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/653248774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:31.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:31 smithi136 ceph-mon[35883]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:31.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:31 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/653248774' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:32.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:32.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:33.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:33 smithi136 ceph-mon[35883]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:33.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:33 smithi063 ceph-mon[28882]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:34.049 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:34.324 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:35.460 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:35 smithi063 ceph-mon[28882]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:35.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:35 smithi136 ceph-mon[35883]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:35.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 -- 172.21.15.63:0/3542621983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 msgr2=0x7ff3280fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:35.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 --2- 172.21.15.63:0/3542621983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff3280fb200 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7ff3180048d0 tx=0x7ff318009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:35.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 -- 172.21.15.63:0/3542621983 shutdown_connections 2024-06-15T00:10:35.847 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 --2- 172.21.15.63:0/3542621983 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff3280fb200 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 --2- 172.21.15.63:0/3542621983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 0x7ff3281015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 -- 172.21.15.63:0/3542621983 >> 172.21.15.63:0/3542621983 conn(0x7ff3280f6dc0 msgr2=0x7ff3280f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 -- 172.21.15.63:0/3542621983 shutdown_connections 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.848+0000 7ff3302a1640 1 -- 172.21.15.63:0/3542621983 wait complete. 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 Processor -- start 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 -- start start 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 0x7ff328112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff328113a50 con 0x7ff328101b70 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff3302a1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32810ce90 con 0x7ff3281011d0 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff32d83c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff32d83c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47826/0 (socket says 172.21.15.63:47826) 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff32d83c640 1 -- 172.21.15.63:0/2794838329 learned_addr learned my addr 172.21.15.63:0/2794838329 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:35.848 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.849+0000 7ff32e03d640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 0x7ff328112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff32d83c640 1 -- 172.21.15.63:0/2794838329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 msgr2=0x7ff328112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff32d83c640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 0x7ff328112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff32d83c640 1 -- 172.21.15.63:0/2794838329 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff318005040 con 0x7ff328101b70 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff32d83c640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff3180048a0 tx=0x7ff318009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff318038070 con 0x7ff328101b70 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff318032d30 con 0x7ff328101b70 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32810d110 con 0x7ff328101b70 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff318009680 con 0x7ff328101b70 2024-06-15T00:10:35.849 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.850+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32810d650 con 0x7ff328101b70 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.851+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3280fc940 con 0x7ff328101b70 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.851+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff318030020 con 0x7ff328101b70 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.852+0000 7ff31effd640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 0x7ff3140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.852+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff3180a2140 con 0x7ff328101b70 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.852+0000 7ff32e03d640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 0x7ff3140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:35.850 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.852+0000 7ff32e03d640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 0x7ff3140678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3240041d0 tx=0x7ff324004330 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:35.853 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:35.856+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff318070720 con 0x7ff328101b70 2024-06-15T00:10:36.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.065+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff3280fa9c0 con 0x7ff328101b70 2024-06-15T00:10:36.062 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.065+0000 7ff31effd640 1 -- 172.21.15.63:0/2794838329 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff318073d40 con 0x7ff328101b70 2024-06-15T00:10:36.062 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:36.062 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 msgr2=0x7ff3140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 0x7ff3140678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff3240041d0 tx=0x7ff324004330 comp rx=0 tx=0).stop 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 msgr2=0x7ff328113450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff3180048a0 tx=0x7ff318009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 shutdown_connections 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff314065430 0x7ff3140678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:36.064 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff328101b70 0x7ff328113450 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:36.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 --2- 172.21.15.63:0/2794838329 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3281011d0 0x7ff328112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:36.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 >> 172.21.15.63:0/2794838329 conn(0x7ff3280f6dc0 msgr2=0x7ff3280f8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:36.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 shutdown_connections 2024-06-15T00:10:36.065 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:36.067+0000 7ff3302a1640 1 -- 172.21.15.63:0/2794838329 wait complete. 2024-06-15T00:10:36.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:36 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2794838329' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:36.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:36 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2794838329' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:37.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:37 smithi136 ceph-mon[35883]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:37.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:37 smithi063 ceph-mon[28882]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:39.448 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:39 smithi063 ceph-mon[28882]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:39.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:39 smithi136 ceph-mon[35883]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:39.721 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:39.988 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:41.333 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:41 smithi063 ceph-mon[28882]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- 172.21.15.63:0/1888620945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f4102c90 msgr2=0x7f58f40fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 --2- 172.21.15.63:0/1888620945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f4102c90 0x7f58f40fc320 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f58e80048d0 tx=0x7f58e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- 172.21.15.63:0/1888620945 shutdown_connections 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 --2- 172.21.15.63:0/1888620945 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f4102c90 0x7f58f40fc320 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 --2- 172.21.15.63:0/1888620945 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f41022f0 0x7f58f41026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- 172.21.15.63:0/1888620945 >> 172.21.15.63:0/1888620945 conn(0x7f58f40f7f00 msgr2=0x7f58f40fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- 172.21.15.63:0/1888620945 shutdown_connections 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- 172.21.15.63:0/1888620945 wait complete. 2024-06-15T00:10:41.518 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 Processor -- start 2024-06-15T00:10:41.519 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.521+0000 7f58fad24640 1 -- start start 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58fad24640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58fad24640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f4102c90 0x7f58f406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58fad24640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f40701f0 con 0x7f58f41022f0 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58fad24640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f4070360 con 0x7f58f4102c90 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53974/0 (socket says 172.21.15.63:53974) 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 -- 172.21.15.63:0/2447707205 learned_addr learned my addr 172.21.15.63:0/2447707205 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f3fff640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f4102c90 0x7f58f406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:41.520 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 -- 172.21.15.63:0/2447707205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f4102c90 msgr2=0x7f58f406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f4102c90 0x7f58f406fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 -- 172.21.15.63:0/2447707205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8005040 con 0x7f58f41022f0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.522+0000 7f58f8ac0640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c830 tx=0x7f58e4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.523+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58e400a7d0 con 0x7f58f41022f0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.523+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58e400ade0 con 0x7f58f41022f0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.523+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f40705f0 con 0x7f58f41022f0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.523+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58e40113a0 con 0x7f58f41022f0 2024-06-15T00:10:41.521 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.523+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f4119630 con 0x7f58f41022f0 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.524+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f40fda60 con 0x7f58f41022f0 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.524+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f58e4012400 con 0x7f58f41022f0 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.525+0000 7f58f17fa640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 0x7f58dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.525+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58e4082620 con 0x7f58f41022f0 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.525+0000 7f58f3fff640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 0x7f58dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:41.522 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.525+0000 7f58f3fff640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 0x7f58dc0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58e802a600 tx=0x7f58e8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:41.525 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.529+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f58e4050b50 con 0x7f58f41022f0 2024-06-15T00:10:41.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:41 smithi136 ceph-mon[35883]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:41.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.740+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f58f40fbb00 con 0x7f58f41022f0 2024-06-15T00:10:41.737 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.740+0000 7f58f17fa640 1 -- 172.21.15.63:0/2447707205 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f58e4054170 con 0x7f58f41022f0 2024-06-15T00:10:41.737 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:41.737 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:41.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 msgr2=0x7f58dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:41.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 0x7f58dc0678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f58e802a600 tx=0x7f58e8035040 comp rx=0 tx=0).stop 2024-06-15T00:10:41.739 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 msgr2=0x7f58f406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f58e400c830 tx=0x7f58e4004430 comp rx=0 tx=0).stop 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 shutdown_connections 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f58dc065430 0x7f58dc0678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58f4102c90 0x7f58f406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 --2- 172.21.15.63:0/2447707205 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f58f41022f0 0x7f58f406f770 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 >> 172.21.15.63:0/2447707205 conn(0x7f58f40f7f00 msgr2=0x7f58f40f8f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 shutdown_connections 2024-06-15T00:10:41.740 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:41.742+0000 7f58fad24640 1 -- 172.21.15.63:0/2447707205 wait complete. 2024-06-15T00:10:42.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:42 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2447707205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:42.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:42 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2447707205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:43.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:43 smithi136 ceph-mon[35883]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:43.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:43 smithi063 ceph-mon[28882]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:45.344 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:45.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:45 smithi136 ceph-mon[35883]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:45.620 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:45.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:45 smithi063 ceph-mon[28882]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/1103292563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc102c60 msgr2=0x7fe1dc0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/1103292563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc102c60 0x7fe1dc0fc2f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe1d00048d0 tx=0x7fe1d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/1103292563 shutdown_connections 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/1103292563 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc102c60 0x7fe1dc0fc2f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/1103292563 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/1103292563 >> 172.21.15.63:0/1103292563 conn(0x7fe1dc0f7e50 msgr2=0x7fe1dc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/1103292563 shutdown_connections 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.169+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/1103292563 wait complete. 2024-06-15T00:10:47.168 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 Processor -- start 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 -- start start 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 0x7fe1dc113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc113ec0 con 0x7fe1dc1022c0 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1e1e6d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc10d270 con 0x7fe1dc102c60 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1db7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1db7fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53982/0 (socket says 172.21.15.63:53982) 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.170+0000 7fe1db7fe640 1 -- 172.21.15.63:0/2163304298 learned_addr learned my addr 172.21.15.63:0/2163304298 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1daffd640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 0x7fe1dc113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1db7fe640 1 -- 172.21.15.63:0/2163304298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 msgr2=0x7fe1dc113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1db7fe640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 0x7fe1dc113830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1db7fe640 1 -- 172.21.15.63:0/2163304298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0005040 con 0x7fe1dc1022c0 2024-06-15T00:10:47.169 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1daffd640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 0x7fe1dc113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1db7fe640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00b790 tx=0x7fe1cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1cc00c700 con 0x7fe1dc1022c0 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1cc00cd10 con 0x7fe1dc1022c0 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc10d550 con 0x7fe1dc1022c0 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe1cc013370 con 0x7fe1dc1022c0 2024-06-15T00:10:47.170 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.171+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc10da70 con 0x7fe1dc1022c0 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.172+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc0fda30 con 0x7fe1dc1022c0 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.173+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fe1cc012070 con 0x7fe1dc1022c0 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.173+0000 7fe1c3fff640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 0x7fe1c40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.173+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe1cc082140 con 0x7fe1dc1022c0 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.173+0000 7fe1daffd640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 0x7fe1c40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:47.171 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.174+0000 7fe1daffd640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 0x7fe1c40678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0007ac0 tx=0x7fe1d0009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:47.174 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.177+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe1cc0505f0 con 0x7fe1dc1022c0 2024-06-15T00:10:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.390+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe1dc109340 con 0x7fe1dc1022c0 2024-06-15T00:10:47.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:47 smithi063 ceph-mon[28882]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:47.387 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:47.387 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.390+0000 7fe1c3fff640 1 -- 172.21.15.63:0/2163304298 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe1cc053c10 con 0x7fe1dc1022c0 2024-06-15T00:10:47.388 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:47.388 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:47.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 msgr2=0x7fe1c40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:47.389 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 0x7fe1c40678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0007ac0 tx=0x7fe1d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 msgr2=0x7fe1dc1132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00b790 tx=0x7fe1cc0091d0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 shutdown_connections 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fe1c4065430 0x7fe1c40678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1dc102c60 0x7fe1dc113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 --2- 172.21.15.63:0/2163304298 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fe1dc1022c0 0x7fe1dc1132f0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 >> 172.21.15.63:0/2163304298 conn(0x7fe1dc0f7e50 msgr2=0x7fe1dc0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 shutdown_connections 2024-06-15T00:10:47.390 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:47.393+0000 7fe1e1e6d640 1 -- 172.21.15.63:0/2163304298 wait complete. 2024-06-15T00:10:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:47 smithi136 ceph-mon[35883]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:47.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:10:48.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:48 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2163304298' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:48.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:48 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2163304298' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:49.515 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:49 smithi063 ceph-mon[28882]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:49.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:49 smithi136 ceph-mon[35883]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:51.024 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:51.303 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:51.426 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:51 smithi063 ceph-mon[28882]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:51.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:51 smithi136 ceph-mon[35883]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:52.856 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 -- 172.21.15.63:0/469071385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 msgr2=0x7ff74c102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 --2- 172.21.15.63:0/469071385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c102690 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7ff73c009c50 tx=0x7ff73c004d10 comp rx=0 tx=0).stop 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 -- 172.21.15.63:0/469071385 shutdown_connections 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 --2- 172.21.15.63:0/469071385 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 0x7ff74c0fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 --2- 172.21.15.63:0/469071385 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c102690 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 -- 172.21.15.63:0/469071385 >> 172.21.15.63:0/469071385 conn(0x7ff74c0f7e50 msgr2=0x7ff74c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 -- 172.21.15.63:0/469071385 shutdown_connections 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 -- 172.21.15.63:0/469071385 wait complete. 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.858+0000 7ff7508db640 1 Processor -- start 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff7508db640 1 -- start start 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff7508db640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff7508db640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 0x7ff74c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff7508db640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c070280 con 0x7ff74c1022c0 2024-06-15T00:10:52.857 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff7508db640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c0703f0 con 0x7ff74c102c60 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47100/0 (socket says 172.21.15.63:47100) 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 -- 172.21.15.63:0/2662429096 learned_addr learned my addr 172.21.15.63:0/2662429096 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff749d9b640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 0x7ff74c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 -- 172.21.15.63:0/2662429096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 msgr2=0x7ff74c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 0x7ff74c06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 -- 172.21.15.63:0/2662429096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c0098b0 con 0x7ff74c1022c0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.859+0000 7ff74a59c640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff73c003400 tx=0x7ff73c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.860+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73c041070 con 0x7ff74c1022c0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.860+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff73c009690 con 0x7ff74c1022c0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.860+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74c119610 con 0x7ff74c1022c0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.860+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff73c03d8c0 con 0x7ff74c1022c0 2024-06-15T00:10:52.858 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.860+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c119990 con 0x7ff74c1022c0 2024-06-15T00:10:52.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.861+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c0fda30 con 0x7ff74c1022c0 2024-06-15T00:10:52.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.861+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7ff73c02f030 con 0x7ff74c1022c0 2024-06-15T00:10:52.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.862+0000 7ff742ffd640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 0x7ff7380678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:52.859 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.862+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff73c0a2ff0 con 0x7ff74c1022c0 2024-06-15T00:10:52.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.862+0000 7ff749d9b640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 0x7ff7380678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:52.860 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.862+0000 7ff749d9b640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 0x7ff7380678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0710b0 tx=0x7ff734009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:52.862 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:52.866+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff73c0714a0 con 0x7ff74c1022c0 2024-06-15T00:10:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.078+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff74c0fbab0 con 0x7ff74c1022c0 2024-06-15T00:10:53.075 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.079+0000 7ff742ffd640 1 -- 172.21.15.63:0/2662429096 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff73c074ac0 con 0x7ff74c1022c0 2024-06-15T00:10:53.076 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:53.076 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 msgr2=0x7ff7380678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 0x7ff7380678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff74c0710b0 tx=0x7ff734009120 comp rx=0 tx=0).stop 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 msgr2=0x7ff74c06f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ff73c003400 tx=0x7ff73c004d10 comp rx=0 tx=0).stop 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 shutdown_connections 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7ff738065430 0x7ff7380678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff74c102c60 0x7ff74c06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 --2- 172.21.15.63:0/2662429096 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7ff74c1022c0 0x7ff74c06f770 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 >> 172.21.15.63:0/2662429096 conn(0x7ff74c0f7e50 msgr2=0x7ff74c0f8e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 shutdown_connections 2024-06-15T00:10:53.078 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:53.081+0000 7ff7508db640 1 -- 172.21.15.63:0/2662429096 wait complete. 2024-06-15T00:10:53.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:53 smithi063 ceph-mon[28882]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:53.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:53 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2662429096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:53.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:53 smithi136 ceph-mon[35883]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:53.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:53 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2662429096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:55.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:55 smithi136 ceph-mon[35883]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:55.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:55 smithi063 ceph-mon[28882]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:56.670 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:10:56.948 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:10:57.331 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:57 smithi063 ceph-mon[28882]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:57.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:57 smithi136 ceph-mon[35883]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:58.469 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 -- 172.21.15.63:0/3571735825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102ab0 msgr2=0x7f459c0fc140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 --2- 172.21.15.63:0/3571735825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102ab0 0x7f459c0fc140 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f45900048d0 tx=0x7f4590009da0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 -- 172.21.15.63:0/3571735825 shutdown_connections 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 --2- 172.21.15.63:0/3571735825 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102ab0 0x7f459c0fc140 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 --2- 172.21.15.63:0/3571735825 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102110 0x7f459c1024e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 -- 172.21.15.63:0/3571735825 >> 172.21.15.63:0/3571735825 conn(0x7f459c0f7ce0 msgr2=0x7f459c0fa0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 -- 172.21.15.63:0/3571735825 shutdown_connections 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 -- 172.21.15.63:0/3571735825 wait complete. 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.471+0000 7f45a179d640 1 Processor -- start 2024-06-15T00:10:58.470 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f45a179d640 1 -- start start 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f45a179d640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f45a179d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102ab0 0x7f459c1132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f45a179d640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c1138f0 con 0x7f459c102110 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f45a179d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f459c10cd30 con 0x7f459c102ab0 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47118/0 (socket says 172.21.15.63:47118) 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 -- 172.21.15.63:0/3913244626 learned_addr learned my addr 172.21.15.63:0/3913244626 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459b7fe640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102ab0 0x7f459c1132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 -- 172.21.15.63:0/3913244626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102ab0 msgr2=0x7f459c1132f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102ab0 0x7f459c1132f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.472+0000 7f459bfff640 1 -- 172.21.15.63:0/3913244626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4590005040 con 0x7f459c102110 2024-06-15T00:10:58.471 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f459bfff640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f458c00c790 tx=0x7f458c0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:58.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f458c00a820 con 0x7f459c102110 2024-06-15T00:10:58.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f458c00ae30 con 0x7f459c102110 2024-06-15T00:10:58.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c10d010 con 0x7f459c102110 2024-06-15T00:10:58.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f458c0113a0 con 0x7f459c102110 2024-06-15T00:10:58.472 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.473+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c10d5b0 con 0x7f459c102110 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.474+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c0fd880 con 0x7f459c102110 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.474+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f458c012400 con 0x7f459c102110 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.475+0000 7f4598ff9640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 0x7f45840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.475+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f458c082580 con 0x7f459c102110 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.475+0000 7f459b7fe640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 0x7f45840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:10:58.473 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.475+0000 7f459b7fe640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 0x7f45840678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f459000b020 tx=0x7f4590035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:10:58.475 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.479+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f458c050c10 con 0x7f459c102110 2024-06-15T00:10:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.696+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f459c109130 con 0x7f459c102110 2024-06-15T00:10:58.693 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.696+0000 7f4598ff9640 1 -- 172.21.15.63:0/3913244626 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f458c054230 con 0x7f459c102110 2024-06-15T00:10:58.693 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:10:58.693 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:10:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 msgr2=0x7f45840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 0x7f45840678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f459000b020 tx=0x7f4590035040 comp rx=0 tx=0).stop 2024-06-15T00:10:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 msgr2=0x7f459c112db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:10:58.695 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f458c00c790 tx=0x7f458c0043c0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 shutdown_connections 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f4584065430 0x7f45840678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f459c102ab0 0x7f459c1132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.698+0000 7f45a179d640 1 --2- 172.21.15.63:0/3913244626 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f459c102110 0x7f459c112db0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.699+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 >> 172.21.15.63:0/3913244626 conn(0x7f459c0f7ce0 msgr2=0x7f459c0f8ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.699+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 shutdown_connections 2024-06-15T00:10:58.696 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:10:58.699+0000 7f45a179d640 1 -- 172.21.15.63:0/3913244626 wait complete. 2024-06-15T00:10:59.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:59 smithi063 ceph-mon[28882]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:59.355 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:10:59 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3913244626' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:10:59.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:59 smithi136 ceph-mon[35883]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:10:59.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:10:59 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3913244626' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:01.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:01 smithi136 ceph-mon[35883]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:01.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:01 smithi063 ceph-mon[28882]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:02.357 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:02.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:11:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:11:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:11:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:11:02.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:02 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:02.620 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:02.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:11:02.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:11:02.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:11:02.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' 2024-06-15T00:11:02.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:02 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:03.357 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:03 smithi063 ceph-mon[28882]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:03.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:03 smithi136 ceph-mon[35883]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 -- 172.21.15.63:0/1594788959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fbe80 msgr2=0x7f7a780fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 --2- 172.21.15.63:0/1594788959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fbe80 0x7f7a780fcd80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c009c50 tx=0x7f7a6c004d10 comp rx=0 tx=0).stop 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 -- 172.21.15.63:0/1594788959 shutdown_connections 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 --2- 172.21.15.63:0/1594788959 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fbe80 0x7f7a780fcd80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 --2- 172.21.15.63:0/1594788959 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a780fb570 0x7f7a780fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 -- 172.21.15.63:0/1594788959 >> 172.21.15.63:0/1594788959 conn(0x7f7a78072110 msgr2=0x7f7a78072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 -- 172.21.15.63:0/1594788959 shutdown_connections 2024-06-15T00:11:04.251 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.252+0000 7f7a7f570640 1 -- 172.21.15.63:0/1594788959 wait complete. 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 Processor -- start 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 -- start start 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a780fbe80 0x7f7a7810e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7810eb80 con 0x7f7a780fb570 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7f570640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7810ecf0 con 0x7f7a780fbe80 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:42406/0 (socket says 172.21.15.63:42406) 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 -- 172.21.15.63:0/3574797284 learned_addr learned my addr 172.21.15.63:0/3574797284 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 -- 172.21.15.63:0/3574797284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a780fbe80 msgr2=0x7f7a7810e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a780fbe80 0x7f7a7810e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 -- 172.21.15.63:0/3574797284 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a6c0098b0 con 0x7f7a780fb570 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.253+0000 7f7a7d30c640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b790 tx=0x7f7a680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.254+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a6800c700 con 0x7f7a780fb570 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.254+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a6800cd10 con 0x7f7a780fb570 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.254+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a78116fd0 con 0x7f7a780fb570 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.254+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7a68013370 con 0x7f7a780fb570 2024-06-15T00:11:04.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.254+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a781174c0 con 0x7f7a780fb570 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.255+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a780fc920 con 0x7f7a780fb570 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.255+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7a68012070 con 0x7f7a780fb570 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.256+0000 7f7a75ffb640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 0x7f7a600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.256+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7a68082000 con 0x7f7a780fb570 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.256+0000 7f7a7cb0b640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 0x7f7a600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:04.254 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.256+0000 7f7a7cb0b640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 0x7f7a600678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c004a30 tx=0x7f7a6c004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:04.256 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.260+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7a68050530 con 0x7f7a780fb570 2024-06-15T00:11:04.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.468+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7a7804b480 con 0x7f7a780fb570 2024-06-15T00:11:04.465 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.468+0000 7f7a75ffb640 1 -- 172.21.15.63:0/3574797284 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7a68053b50 con 0x7f7a780fb570 2024-06-15T00:11:04.465 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:04.465 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:04.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 msgr2=0x7f7a600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:04.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 0x7f7a600678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f7a6c004a30 tx=0x7f7a6c004d10 comp rx=0 tx=0).stop 2024-06-15T00:11:04.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 msgr2=0x7f7a7810df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:04.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f7a6800b790 tx=0x7f7a680091d0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.467 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 shutdown_connections 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7a60065430 0x7f7a600678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a780fbe80 0x7f7a7810e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 --2- 172.21.15.63:0/3574797284 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7a780fb570 0x7f7a7810df60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.470+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 >> 172.21.15.63:0/3574797284 conn(0x7f7a78072110 msgr2=0x7f7a780f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.471+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 shutdown_connections 2024-06-15T00:11:04.468 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:04.471+0000 7f7a7f570640 1 -- 172.21.15.63:0/3574797284 wait complete. 2024-06-15T00:11:05.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:05 smithi063 ceph-mon[28882]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:05.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:05 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3574797284' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:05.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:05 smithi136 ceph-mon[35883]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:05.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:05 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3574797284' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:07.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:07 smithi136 ceph-mon[35883]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:07.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:07 smithi063 ceph-mon[28882]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:08.048 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:08.326 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:09.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:09 smithi136 ceph-mon[35883]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:09.597 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:09 smithi063 ceph-mon[28882]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:09.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 -- 172.21.15.63:0/296983629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 msgr2=0x7f0e100fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:09.922 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 --2- 172.21.15.63:0/296983629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e100fcd80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f0df80048d0 tx=0x7f0df8009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 -- 172.21.15.63:0/296983629 shutdown_connections 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 --2- 172.21.15.63:0/296983629 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e100fcd80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 --2- 172.21.15.63:0/296983629 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 0x7f0e100fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 -- 172.21.15.63:0/296983629 >> 172.21.15.63:0/296983629 conn(0x7f0e10072110 msgr2=0x7f0e10072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 -- 172.21.15.63:0/296983629 shutdown_connections 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.924+0000 7f0e1523c640 1 -- 172.21.15.63:0/296983629 wait complete. 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 Processor -- start 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 -- start start 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 0x7f0e10112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e10113a50 con 0x7f0e100fbe80 2024-06-15T00:11:09.923 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e1523c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e1010ce90 con 0x7f0e100fb570 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40118/0 (socket says 172.21.15.63:40118) 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 -- 172.21.15.63:0/4078433533 learned_addr learned my addr 172.21.15.63:0/4078433533 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0ed9d640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 0x7f0e10112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 -- 172.21.15.63:0/4078433533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 msgr2=0x7f0e10112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 0x7f0e10112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.925+0000 7f0e0e59c640 1 -- 172.21.15.63:0/4078433533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8005040 con 0x7f0e100fbe80 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e0e59c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0df80048a0 tx=0x7f0df8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0df8038070 con 0x7f0e100fbe80 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e1010d110 con 0x7f0e100fbe80 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0df802cef0 con 0x7f0e100fbe80 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0df8009640 con 0x7f0e100fbe80 2024-06-15T00:11:09.924 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.926+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e1010d650 con 0x7f0e100fbe80 2024-06-15T00:11:09.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.927+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e100fc920 con 0x7f0e100fbe80 2024-06-15T00:11:09.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.927+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0df8030020 con 0x7f0e100fbe80 2024-06-15T00:11:09.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.928+0000 7f0e077fe640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:09.925 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.928+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0df80a2930 con 0x7f0e100fbe80 2024-06-15T00:11:09.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.928+0000 7f0e0ed9d640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:09.926 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.928+0000 7f0e0ed9d640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 0x7f0dfc0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0e08007fd0 tx=0x7f0e08009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:09.928 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:09.932+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0df8070de0 con 0x7f0e100fbe80 2024-06-15T00:11:10.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.141+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e1010e580 con 0x7f0e100fbe80 2024-06-15T00:11:10.138 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.142+0000 7f0e077fe640 1 -- 172.21.15.63:0/4078433533 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0df8074400 con 0x7f0e100fbe80 2024-06-15T00:11:10.139 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:10.139 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 msgr2=0x7f0dfc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:10.140 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 0x7f0dfc0678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0e08007fd0 tx=0x7f0e08009e90 comp rx=0 tx=0).stop 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 msgr2=0x7f0e10113450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0df80048a0 tx=0x7f0df8009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 shutdown_connections 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0dfc065430 0x7f0dfc0678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0e100fbe80 0x7f0e10113450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 --2- 172.21.15.63:0/4078433533 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e100fb570 0x7f0e10112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 >> 172.21.15.63:0/4078433533 conn(0x7f0e10072110 msgr2=0x7f0e10065570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 shutdown_connections 2024-06-15T00:11:10.141 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:10.144+0000 7f0e1523c640 1 -- 172.21.15.63:0/4078433533 wait complete. 2024-06-15T00:11:10.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:10 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/4078433533' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:10.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:10 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/4078433533' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:11.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:11 smithi136 ceph-mon[35883]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:11.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:11 smithi063 ceph-mon[28882]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:13.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:13 smithi136 ceph-mon[35883]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:13.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:13 smithi063 ceph-mon[28882]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:13.872 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:14.161 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:15.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:15 smithi136 ceph-mon[35883]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:15.583 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:15 smithi063 ceph-mon[28882]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:15.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.796+0000 7f3e39b09640 1 -- 172.21.15.63:0/172867157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 msgr2=0x7f3e34106310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:15.793 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.796+0000 7f3e39b09640 1 --2- 172.21.15.63:0/172867157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e34106310 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f3e28004900 tx=0x7f3e28009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 -- 172.21.15.63:0/172867157 shutdown_connections 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 --2- 172.21.15.63:0/172867157 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e34106310 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 --2- 172.21.15.63:0/172867157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 0x7f3e340fd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 -- 172.21.15.63:0/172867157 >> 172.21.15.63:0/172867157 conn(0x7f3e340f6d60 msgr2=0x7f3e340f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 -- 172.21.15.63:0/172867157 shutdown_connections 2024-06-15T00:11:15.794 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 -- 172.21.15.63:0/172867157 wait complete. 2024-06-15T00:11:15.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.797+0000 7f3e39b09640 1 Processor -- start 2024-06-15T00:11:15.795 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e39b09640 1 -- start start 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e39b09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 0x7f3e340fb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e39b09640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e39b09640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340fbbe0 con 0x7f3e340fdc40 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e39b09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e340fbd50 con 0x7f3e340fd2a0 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:40148/0 (socket says 172.21.15.63:40148) 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 -- 172.21.15.63:0/2310443349 learned_addr learned my addr 172.21.15.63:0/2310443349 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e337fe640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 0x7f3e340fb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 -- 172.21.15.63:0/2310443349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 msgr2=0x7f3e340fb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 0x7f3e340fb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 -- 172.21.15.63:0/2310443349 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28005040 con 0x7f3e340fdc40 2024-06-15T00:11:15.796 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.798+0000 7f3e32ffd640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e28004960 tx=0x7f3e28009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.799+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e28038070 con 0x7f3e340fdc40 2024-06-15T00:11:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.799+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e2802ae90 con 0x7f3e340fdc40 2024-06-15T00:11:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.799+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e340fbf80 con 0x7f3e340fdc40 2024-06-15T00:11:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.799+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3e28009550 con 0x7f3e340fdc40 2024-06-15T00:11:15.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.799+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e34119660 con 0x7f3e340fdc40 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.800+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e34066030 con 0x7f3e340fdc40 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.800+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f3e28030020 con 0x7f3e340fdc40 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.801+0000 7f3e1bfff640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 0x7f3e1c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.801+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3e280a2850 con 0x7f3e340fdc40 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.801+0000 7f3e337fe640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 0x7f3e1c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:15.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.801+0000 7f3e337fe640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 0x7f3e1c0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e24004ed0 tx=0x7f3e2400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:15.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:15.805+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3e28070d80 con 0x7f3e340fdc40 2024-06-15T00:11:16.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.015+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e3404b480 con 0x7f3e340fdc40 2024-06-15T00:11:16.013 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.016+0000 7f3e1bfff640 1 -- 172.21.15.63:0/2310443349 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3e280743a0 con 0x7f3e340fdc40 2024-06-15T00:11:16.013 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:16.013 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 msgr2=0x7f3e1c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 0x7f3e1c0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f3e24004ed0 tx=0x7f3e2400b040 comp rx=0 tx=0).stop 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 msgr2=0x7f3e340fb6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f3e28004960 tx=0x7f3e28009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 shutdown_connections 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f3e1c065430 0x7f3e1c0678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f3e340fdc40 0x7f3e340fb6a0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 --2- 172.21.15.63:0/2310443349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e340fd2a0 0x7f3e340fb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 >> 172.21.15.63:0/2310443349 conn(0x7f3e340f6d60 msgr2=0x7f3e340f82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 shutdown_connections 2024-06-15T00:11:16.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:16.018+0000 7f3e39b09640 1 -- 172.21.15.63:0/2310443349 wait complete. 2024-06-15T00:11:16.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:16 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/2310443349' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:16.668 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:16 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/2310443349' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:17 smithi136 ceph-mon[35883]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:17.583 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:17 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:17.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:17 smithi063 ceph-mon[28882]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:17.678 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:17 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:19.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:19 smithi136 ceph-mon[35883]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:19.677 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:19 smithi063 ceph-mon[28882]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:19.694 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:19.971 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:20.405 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:20 smithi063 ceph-mon[28882]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:20.582 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:20 smithi136 ceph-mon[35883]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:21.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 -- 172.21.15.63:0/1146992066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb410ccd0 msgr2=0x7febb410d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 --2- 172.21.15.63:0/1146992066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb410ccd0 0x7febb410d0a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7feba40048d0 tx=0x7feba4009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 -- 172.21.15.63:0/1146992066 shutdown_connections 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 --2- 172.21.15.63:0/1146992066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb4107490 0x7febb4107900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 --2- 172.21.15.63:0/1146992066 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb410ccd0 0x7febb410d0a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.553+0000 7febb9a62640 1 -- 172.21.15.63:0/1146992066 >> 172.21.15.63:0/1146992066 conn(0x7febb40672a0 msgr2=0x7febb40676a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 -- 172.21.15.63:0/1146992066 shutdown_connections 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 -- 172.21.15.63:0/1146992066 wait complete. 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 Processor -- start 2024-06-15T00:11:21.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 -- start start 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 0x7febb411de80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb411e4d0 con 0x7febb4107490 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb9a62640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febb41178e0 con 0x7febb410ccd0 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.554+0000 7febb37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47340/0 (socket says 172.21.15.63:47340) 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 -- 172.21.15.63:0/110477047 learned_addr learned my addr 172.21.15.63:0/110477047 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb2ffd640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 0x7febb411de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 -- 172.21.15.63:0/110477047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 msgr2=0x7febb411de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 0x7febb411de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 -- 172.21.15.63:0/110477047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feba4005040 con 0x7febb4107490 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb2ffd640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 0x7febb411de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:11:21.553 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb37fe640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feba4000c00 tx=0x7feba4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:21.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba4038070 con 0x7febb4107490 2024-06-15T00:11:21.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febb4117b60 con 0x7febb4107490 2024-06-15T00:11:21.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feba40096f0 con 0x7febb4107490 2024-06-15T00:11:21.554 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feba4032da0 con 0x7febb4107490 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.555+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febb41180a0 con 0x7febb4107490 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.556+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febb41084a0 con 0x7febb4107490 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.557+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7feba4030020 con 0x7febb4107490 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.557+0000 7feb9bfff640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 0x7feb9c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.557+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feba40a2a80 con 0x7febb4107490 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.557+0000 7febb2ffd640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 0x7feb9c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:21.555 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.558+0000 7febb2ffd640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 0x7feb9c0678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febb4118bc0 tx=0x7feba8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:21.558 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.561+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7feba4070f30 con 0x7febb4107490 2024-06-15T00:11:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.768+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7febb4118fe0 con 0x7febb4107490 2024-06-15T00:11:21.765 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.768+0000 7feb9bfff640 1 -- 172.21.15.63:0/110477047 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7feba4074550 con 0x7febb4107490 2024-06-15T00:11:21.765 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:21.765 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 msgr2=0x7feb9c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 0x7feb9c0678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7febb4118bc0 tx=0x7feba8009040 comp rx=0 tx=0).stop 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 msgr2=0x7febb411d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7feba4000c00 tx=0x7feba4009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 shutdown_connections 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7feb9c065430 0x7feb9c0678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.767 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febb410ccd0 0x7febb411de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 --2- 172.21.15.63:0/110477047 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7febb4107490 0x7febb411d940 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.770+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 >> 172.21.15.63:0/110477047 conn(0x7febb40672a0 msgr2=0x7febb41111f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.771+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 shutdown_connections 2024-06-15T00:11:21.768 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:21.771+0000 7febb9a62640 1 -- 172.21.15.63:0/110477047 wait complete. 2024-06-15T00:11:22.082 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:21 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/110477047' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:22.122 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:21 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/110477047' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:23.082 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:22 smithi136 ceph-mon[35883]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:23.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:22 smithi063 ceph-mon[28882]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:25.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:25 smithi136 ceph-mon[35883]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:25.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:25 smithi063 ceph-mon[28882]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:25.429 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:25.705 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:27.143 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:27 smithi063 ceph-mon[28882]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 -- 172.21.15.63:0/784941353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 msgr2=0x7f0c300fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/784941353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c300fc2f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0048d0 tx=0x7f0c2c009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 -- 172.21.15.63:0/784941353 shutdown_connections 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/784941353 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c300fc2f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/784941353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 -- 172.21.15.63:0/784941353 >> 172.21.15.63:0/784941353 conn(0x7f0c300f7e50 msgr2=0x7f0c300fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 -- 172.21.15.63:0/784941353 shutdown_connections 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.334+0000 7f0c37b8f640 1 -- 172.21.15.63:0/784941353 wait complete. 2024-06-15T00:11:27.332 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 Processor -- start 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 -- start start 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30113db0 con 0x7f0c30102c60 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c37b8f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3010d1f0 con 0x7f0c301022c0 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:47360/0 (socket says 172.21.15.63:47360) 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 -- 172.21.15.63:0/819731214 learned_addr learned my addr 172.21.15.63:0/819731214 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3592b640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 -- 172.21.15.63:0/819731214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 msgr2=0x7f0c30113270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30113270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3512a640 1 -- 172.21.15.63:0/819731214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c2c005040 con 0x7f0c30102c60 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.335+0000 7f0c3592b640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:11:27.333 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c3512a640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0048a0 tx=0x7f0c2c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:27.334 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:27 smithi136 ceph-mon[35883]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2c038070 con 0x7f0c30102c60 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c3010d470 con 0x7f0c30102c60 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c2c032d30 con 0x7f0c30102c60 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c2c009680 con 0x7f0c30102c60 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.336+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3010d9b0 con 0x7f0c30102c60 2024-06-15T00:11:27.335 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.337+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c300fda30 con 0x7f0c30102c60 2024-06-15T00:11:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.337+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f0c2c030020 con 0x7f0c30102c60 2024-06-15T00:11:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.338+0000 7f0c267fc640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 0x7f0c1c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.338+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0c2c0a2300 con 0x7f0c30102c60 2024-06-15T00:11:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.338+0000 7f0c3592b640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 0x7f0c1c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:27.336 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.338+0000 7f0c3592b640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 0x7f0c1c0678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0c20004c60 tx=0x7f0c20009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:27.338 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.342+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0c2c0707b0 con 0x7f0c30102c60 2024-06-15T00:11:27.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.551+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0c301092c0 con 0x7f0c30102c60 2024-06-15T00:11:27.548 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.552+0000 7f0c267fc640 1 -- 172.21.15.63:0/819731214 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0c2c073dd0 con 0x7f0c30102c60 2024-06-15T00:11:27.549 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:27.549 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 msgr2=0x7f0c1c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 0x7f0c1c0678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0c20004c60 tx=0x7f0c20009040 comp rx=0 tx=0).stop 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 msgr2=0x7f0c301137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0048a0 tx=0x7f0c2c009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 shutdown_connections 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f0c1c065430 0x7f0c1c0678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f0c30102c60 0x7f0c301137b0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 --2- 172.21.15.63:0/819731214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c301022c0 0x7f0c30113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 >> 172.21.15.63:0/819731214 conn(0x7f0c300f7e50 msgr2=0x7f0c300f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:27.551 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 shutdown_connections 2024-06-15T00:11:27.552 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:27.554+0000 7f0c37b8f640 1 -- 172.21.15.63:0/819731214 wait complete. 2024-06-15T00:11:28.138 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:28 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/819731214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:28.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:28 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/819731214' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:29.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:29 smithi136 ceph-mon[35883]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:29.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:29 smithi063 ceph-mon[28882]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:31.165 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:31.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:31 smithi136 ceph-mon[35883]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:31.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:31 smithi063 ceph-mon[28882]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:31.443 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:32.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:32 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:32.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:32 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:33.040 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 -- 172.21.15.63:0/1758980791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 msgr2=0x7fac7c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 --2- 172.21.15.63:0/1758980791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c102670 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fac740048d0 tx=0x7fac74009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 -- 172.21.15.63:0/1758980791 shutdown_connections 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 --2- 172.21.15.63:0/1758980791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 0x7fac7c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 --2- 172.21.15.63:0/1758980791 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c102670 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 -- 172.21.15.63:0/1758980791 >> 172.21.15.63:0/1758980791 conn(0x7fac7c0f7e50 msgr2=0x7fac7c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 -- 172.21.15.63:0/1758980791 shutdown_connections 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.024+0000 7fac8447a640 1 -- 172.21.15.63:0/1758980791 wait complete. 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 Processor -- start 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 -- start start 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 0x7fac7c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:33.041 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c0701f0 con 0x7fac7c1022a0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac8447a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c070360 con 0x7fac7c102c40 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35886/0 (socket says 172.21.15.63:35886) 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 -- 172.21.15.63:0/3211691109 learned_addr learned my addr 172.21.15.63:0/3211691109 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac81a15640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 0x7fac7c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 -- 172.21.15.63:0/3211691109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 msgr2=0x7fac7c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 0x7fac7c06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 -- 172.21.15.63:0/3211691109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac74005040 con 0x7fac7c1022a0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.025+0000 7fac82216640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac740048a0 tx=0x7fac74009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.026+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac74038070 con 0x7fac7c1022a0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.026+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c070590 con 0x7fac7c1022a0 2024-06-15T00:11:33.042 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.026+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac74032d30 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.026+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac74009680 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.026+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c119500 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.027+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c0fda10 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.027+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7fac74030020 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.028+0000 7fac7affd640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 0x7fac640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.028+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fac740a2250 con 0x7fac7c1022a0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.028+0000 7fac81a15640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 0x7fac640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.028+0000 7fac81a15640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 0x7fac640678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0710b0 tx=0x7fac70009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:33.043 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.034+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fac740707b0 con 0x7fac7c1022a0 2024-06-15T00:11:33.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:33 smithi063 ceph-mon[28882]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:33.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.253+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fac7c0fba90 con 0x7fac7c1022a0 2024-06-15T00:11:33.250 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.253+0000 7fac7affd640 1 -- 172.21.15.63:0/3211691109 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fac74073dd0 con 0x7fac7c1022a0 2024-06-15T00:11:33.250 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:33.250 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:33.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 msgr2=0x7fac640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:33.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 0x7fac640678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0710b0 tx=0x7fac70009040 comp rx=0 tx=0).stop 2024-06-15T00:11:33.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 msgr2=0x7fac7c06f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:33.252 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fac740048a0 tx=0x7fac74009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 shutdown_connections 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7fac64065430 0x7fac640678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac7c102c40 0x7fac7c06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 --2- 172.21.15.63:0/3211691109 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7fac7c1022a0 0x7fac7c06f770 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 >> 172.21.15.63:0/3211691109 conn(0x7fac7c0f7e50 msgr2=0x7fac7c0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 shutdown_connections 2024-06-15T00:11:33.253 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:33.256+0000 7fac8447a640 1 -- 172.21.15.63:0/3211691109 wait complete. 2024-06-15T00:11:33.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:33 smithi136 ceph-mon[35883]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:34.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:34 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3211691109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:34.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:34 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3211691109' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:35.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:35 smithi136 ceph-mon[35883]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:35.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:35 smithi063 ceph-mon[28882]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:36.914 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:37.193 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:37.229 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:37 smithi063 ceph-mon[28882]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:37.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:37 smithi136 ceph-mon[35883]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 -- 172.21.15.63:0/4181570682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79381022f0 msgr2=0x7f79381026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:38.797 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 --2- 172.21.15.63:0/4181570682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79381022f0 0x7f79381026c0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f79280048d0 tx=0x7f7928009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 -- 172.21.15.63:0/4181570682 shutdown_connections 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 --2- 172.21.15.63:0/4181570682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7938102c90 0x7f79380fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 --2- 172.21.15.63:0/4181570682 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f79381022f0 0x7f79381026c0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 -- 172.21.15.63:0/4181570682 >> 172.21.15.63:0/4181570682 conn(0x7f7938070a40 msgr2=0x7f7938070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 -- 172.21.15.63:0/4181570682 shutdown_connections 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 -- 172.21.15.63:0/4181570682 wait complete. 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.799+0000 7f793d2b3640 1 Processor -- start 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f793d2b3640 1 -- start start 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f793d2b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79381022f0 0x7f7938113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f793d2b3640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f793d2b3640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7938113b40 con 0x7f7938102c90 2024-06-15T00:11:38.798 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f793d2b3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793810cf80 con 0x7f79381022f0 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:35906/0 (socket says 172.21.15.63:35906) 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 -- 172.21.15.63:0/3025959633 learned_addr learned my addr 172.21.15.63:0/3025959633 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f7936ffd640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79381022f0 0x7f7938113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 -- 172.21.15.63:0/3025959633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79381022f0 msgr2=0x7f7938113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79381022f0 0x7f7938113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 -- 172.21.15.63:0/3025959633 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7928005040 con 0x7f7938102c90 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.800+0000 7f79367fc640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f792000c760 tx=0x7f792000cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.801+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f792000a820 con 0x7f7938102c90 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.801+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793810d260 con 0x7f7938102c90 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.801+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f792000ae30 con 0x7f7938102c90 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.801+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f79200113a0 con 0x7f7938102c90 2024-06-15T00:11:38.799 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.801+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793810d780 con 0x7f7938102c90 2024-06-15T00:11:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.802+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7938065510 con 0x7f7938102c90 2024-06-15T00:11:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.802+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f7920012400 con 0x7f7938102c90 2024-06-15T00:11:38.800 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.802+0000 7f792f7fe640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 0x7f79240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.803+0000 7f7936ffd640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 0x7f79240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.803+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f79200826a0 con 0x7f7938102c90 2024-06-15T00:11:38.801 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.803+0000 7f7936ffd640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 0x7f79240678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7928004540 tx=0x7f7928035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:38.803 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:38.807+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7920050c90 con 0x7f7938102c90 2024-06-15T00:11:39.014 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.018+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7938074960 con 0x7f7938102c90 2024-06-15T00:11:39.015 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.018+0000 7f792f7fe640 1 -- 172.21.15.63:0/3025959633 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f79200542b0 con 0x7f7938102c90 2024-06-15T00:11:39.015 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:39.015 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:39.016 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 msgr2=0x7f79240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 0x7f79240678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7928004540 tx=0x7f7928035040 comp rx=0 tx=0).stop 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 msgr2=0x7f7938113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f792000c760 tx=0x7f792000cb60 comp rx=0 tx=0).stop 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 shutdown_connections 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f7924065430 0x7f79240678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f7938102c90 0x7f7938113540 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 --2- 172.21.15.63:0/3025959633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79381022f0 0x7f7938113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 >> 172.21.15.63:0/3025959633 conn(0x7f7938070a40 msgr2=0x7f79380f9470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 shutdown_connections 2024-06-15T00:11:39.017 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:39.020+0000 7f793d2b3640 1 -- 172.21.15.63:0/3025959633 wait complete. 2024-06-15T00:11:39.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:39 smithi136 ceph-mon[35883]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:39.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:39 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/3025959633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:39.342 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:39 smithi063 ceph-mon[28882]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:39.342 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:39 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/3025959633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:41.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:41 smithi136 ceph-mon[35883]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:41.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:41 smithi063 ceph-mon[28882]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:42.640 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:42.918 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:43.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:43 smithi136 ceph-mon[35883]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:43.346 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:43 smithi063 ceph-mon[28882]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.526+0000 7f232c6b9640 1 -- 172.21.15.63:0/2724423697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101310 msgr2=0x7f23241016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.526+0000 7f232c6b9640 1 --2- 172.21.15.63:0/2724423697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101310 0x7f23241016e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f23200048d0 tx=0x7f2320009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- 172.21.15.63:0/2724423697 shutdown_connections 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 --2- 172.21.15.63:0/2724423697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101cb0 0x7f23240fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 --2- 172.21.15.63:0/2724423697 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101310 0x7f23241016e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.536 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- 172.21.15.63:0/2724423697 >> 172.21.15.63:0/2724423697 conn(0x7f23240f6d90 msgr2=0x7f23240f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- 172.21.15.63:0/2724423697 shutdown_connections 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- 172.21.15.63:0/2724423697 wait complete. 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 Processor -- start 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- start start 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101310 0x7f232410e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232410ebc0 con 0x7f2324101cb0 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.527+0000 7f232c6b9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232410ed30 con 0x7f2324101310 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:53148/0 (socket says 172.21.15.63:53148) 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 -- 172.21.15.63:0/168000496 learned_addr learned my addr 172.21.15.63:0/168000496 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f232a455640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101310 0x7f232410e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:44.537 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 -- 172.21.15.63:0/168000496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101310 msgr2=0x7f232410e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101310 0x7f232410e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 -- 172.21.15.63:0/168000496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320005040 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2329c54640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f231800c810 tx=0x7f23180043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f231800a7a0 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f231800adb0 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23241170a0 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2318011380 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.528+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2324117530 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.529+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23240fa9f0 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.530+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f2318012420 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.530+0000 7f2316ffd640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 0x7f23100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.530+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23180828a0 con 0x7f2324101cb0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.530+0000 7f232a455640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 0x7f23100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:44.538 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.531+0000 7f232a455640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 0x7f23100678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f232002a600 tx=0x7f2320035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:44.539 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.534+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2318050dd0 con 0x7f2324101cb0 2024-06-15T00:11:44.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.748+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2324065f60 con 0x7f2324101cb0 2024-06-15T00:11:44.745 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.749+0000 7f2316ffd640 1 -- 172.21.15.63:0/168000496 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f23180543f0 con 0x7f2324101cb0 2024-06-15T00:11:44.745 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:44.745 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:44.747 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 msgr2=0x7f23100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 0x7f23100678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f232002a600 tx=0x7f2320035040 comp rx=0 tx=0).stop 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 msgr2=0x7f232410e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f231800c810 tx=0x7f23180043c0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 shutdown_connections 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f2310065430 0x7f23100678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f2324101cb0 0x7f232410e570 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 --2- 172.21.15.63:0/168000496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2324101310 0x7f232410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 >> 172.21.15.63:0/168000496 conn(0x7f23240f6d90 msgr2=0x7f23240f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 shutdown_connections 2024-06-15T00:11:44.748 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:44.751+0000 7f232c6b9640 1 -- 172.21.15.63:0/168000496 wait complete. 2024-06-15T00:11:45.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:45 smithi136 ceph-mon[35883]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:45.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:45 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/168000496' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:45.423 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:45 smithi063 ceph-mon[28882]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:45.423 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:45 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/168000496' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:47.323 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:47 smithi063 ceph-mon[28882]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:47.323 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:47 smithi063 ceph-mon[28882]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:47.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:47 smithi136 ceph-mon[35883]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:47.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:47 smithi136 ceph-mon[35883]: from='mgr.14212 172.21.15.63:0/2774621988' entity='mgr.smithi063.dhizav' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:11:48.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:11:48.728 INFO:teuthology.orchestra.run.smithi063.stderr:Inferring config /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/mon.smithi063/config 2024-06-15T00:11:49.159 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:49 smithi063 ceph-mon[28882]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:49.332 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:49 smithi136 ceph-mon[35883]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 -- 172.21.15.63:0/4024378645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53280fbaf0 msgr2=0x7f53280fbf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 --2- 172.21.15.63:0/4024378645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53280fbaf0 0x7f53280fbf60 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f53100048d0 tx=0x7f5310009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 -- 172.21.15.63:0/4024378645 shutdown_connections 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 --2- 172.21.15.63:0/4024378645 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53280fbaf0 0x7f53280fbf60 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 --2- 172.21.15.63:0/4024378645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53281001d0 0x7f53281005a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 -- 172.21.15.63:0/4024378645 >> 172.21.15.63:0/4024378645 conn(0x7f5328070a40 msgr2=0x7f5328070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 -- 172.21.15.63:0/4024378645 shutdown_connections 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.234+0000 7f532d625640 1 -- 172.21.15.63:0/4024378645 wait complete. 2024-06-15T00:11:50.232 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 Processor -- start 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 -- start start 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 0x7f532810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 -- --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532810eb70 con 0x7f53281001d0 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f532d625640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532810ece0 con 0x7f53280fbaf0 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 --2- >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.63:3300/0 says I am v2:172.21.15.63:58266/0 (socket says 172.21.15.63:58266) 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 -- 172.21.15.63:0/1337393512 learned_addr learned my addr 172.21.15.63:0/1337393512 (peer_addr_for_me v2:172.21.15.63:0/0) 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f5326ffd640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 0x7f532810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 -- 172.21.15.63:0/1337393512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 msgr2=0x7f532810dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 0x7f532810dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.233 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f53267fc640 1 -- 172.21.15.63:0/1337393512 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310005040 con 0x7f53281001d0 2024-06-15T00:11:50.234 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.235+0000 7f5326ffd640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 0x7f532810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f53267fc640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5310008fc0 tx=0x7f5310009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5310038070 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5310032880 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f532810bdd0 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5310009480 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.236+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f532810c290 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.237+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53280fcb00 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.237+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 4 ==== mgrmap(e 19) v1 ==== 83745+0+0 (secure 0 0 0) 0x7f5310030020 con 0x7f53281001d0 2024-06-15T00:11:50.235 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.238+0000 7f531f7fe640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 0x7f53140678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:11:50.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.238+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f53100a2170 con 0x7f53281001d0 2024-06-15T00:11:50.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.238+0000 7f5326ffd640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 0x7f53140678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:11:50.236 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.238+0000 7f5326ffd640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 0x7f53140678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5318004ab0 tx=0x7f531800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:11:50.238 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.242+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f53100706a0 con 0x7f53281001d0 2024-06-15T00:11:50.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.454+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 --> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5328108330 con 0x7f53281001d0 2024-06-15T00:11:50.451 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.454+0000 7f531f7fe640 1 -- 172.21.15.63:0/1337393512 <== mon.0 v2:172.21.15.63:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5310073cc0 con 0x7f53281001d0 2024-06-15T00:11:50.451 INFO:teuthology.orchestra.run.smithi063.stdout: 2024-06-15T00:11:50.451 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 msgr2=0x7f53140678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 0x7f53140678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5318004ab0 tx=0x7f531800b040 comp rx=0 tx=0).stop 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 msgr2=0x7f532810e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5310008fc0 tx=0x7f5310009da0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 shutdown_connections 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:6800/3568086764,v1:172.21.15.63:6801/3568086764] conn(0x7f5314065430 0x7f53140678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] conn(0x7f53281001d0 0x7f532810e520 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 --2- 172.21.15.63:0/1337393512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53280fbaf0 0x7f532810dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 >> 172.21.15.63:0/1337393512 conn(0x7f5328070a40 msgr2=0x7f53280f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 shutdown_connections 2024-06-15T00:11:50.453 INFO:teuthology.orchestra.run.smithi063.stderr:2024-06-15T00:11:50.456+0000 7f532d625640 1 -- 172.21.15.63:0/1337393512 wait complete. 2024-06-15T00:11:51.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 ceph-mon[28882]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:51.177 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 ceph-mon[28882]: from='client.? 172.21.15.63:0/1337393512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:51.210 INFO:tasks.cephadm:Teardown begin 2024-06-15T00:11:51.213 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:11:51.216 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:11:51.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:11:51.282 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-15T00:11:51.282 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:11:51.307 DEBUG:teuthology.orchestra.run.smithi136:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:11:51.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:51 smithi136 ceph-mon[35883]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:11:51.333 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:51 smithi136 ceph-mon[35883]: from='client.? 172.21.15.63:0/1337393512' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:11:51.341 INFO:tasks.cephadm:Stopping all daemons... 2024-06-15T00:11:51.341 INFO:tasks.cephadm.mon.smithi063:Stopping mon.smithi063... 2024-06-15T00:11:51.341 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063 2024-06-15T00:11:51.650 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 systemd[1]: Stopping Ceph mon.smithi063 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:11:51.927 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063[28878]: 2024-06-15T00:11:51.652+0000 7f14b7b32640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi063 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:11:51.927 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063[28878]: 2024-06-15T00:11:51.652+0000 7f14b7b32640 -1 mon.smithi063@0(leader) e2 *** Got Signal Terminated *** 2024-06-15T00:11:51.927 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:51 smithi063 podman[79644]: 2024-06-15 00:11:51.877851325 +0000 UTC m=+0.321923713 container died 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:11:52.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:52 smithi063 podman[79644]: 2024-06-15 00:11:52.361156455 +0000 UTC m=+0.805228842 container cleanup 5b226ed6e02e1e3ac34107bdc198dc7010f1b6cc3b035a38363670cd5d855ba7 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:11:52.427 INFO:journalctl@ceph.mon.smithi063.smithi063.stdout:Jun 15 00:11:52 smithi063 bash[79644]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi063 2024-06-15T00:11:52.744 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi063.service' 2024-06-15T00:11:52.805 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:11:52.805 INFO:tasks.cephadm.mon.smithi063:Stopped mon.smithi063 2024-06-15T00:11:52.805 INFO:tasks.cephadm.mon.smithi136:Stopping mon.smithi136... 2024-06-15T00:11:52.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi136 2024-06-15T00:11:53.082 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:52 smithi136 systemd[1]: Stopping Ceph mon.smithi136 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:11:53.387 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:53 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi136[35879]: 2024-06-15T00:11:53.115+0000 7f86d1a9d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi136 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:11:53.388 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:53 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi136[35879]: 2024-06-15T00:11:53.115+0000 7f86d1a9d640 -1 mon.smithi136@1(peon) e2 *** Got Signal Terminated *** 2024-06-15T00:11:53.388 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:53 smithi136 podman[59406]: 2024-06-15 00:11:53.126914958 +0000 UTC m=+0.114959835 container died b8c820eb48a5060dbda2675b5781d716d254c62c7117deaa0eb334139daf762b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi136, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:11:53.832 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:53 smithi136 podman[59406]: 2024-06-15 00:11:53.58323837 +0000 UTC m=+0.571283288 container cleanup b8c820eb48a5060dbda2675b5781d716d254c62c7117deaa0eb334139daf762b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi136, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_BRANCH=HEAD) 2024-06-15T00:11:53.832 INFO:journalctl@ceph.mon.smithi136.smithi136.stdout:Jun 15 00:11:53 smithi136 bash[59406]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-mon-smithi136 2024-06-15T00:11:54.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@mon.smithi136.service' 2024-06-15T00:11:54.057 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:11:54.058 INFO:tasks.cephadm.mon.smithi136:Stopped mon.smithi136 2024-06-15T00:11:54.058 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-15T00:11:54.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.0 2024-06-15T00:11:54.370 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:11:54 smithi063 systemd[1]: Stopping Ceph osd.0 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:11:54.677 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:11:54 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0[46684]: 2024-06-15T00:11:54.373+0000 7fcdd7287640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:11:54.677 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:11:54 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0[46684]: 2024-06-15T00:11:54.373+0000 7fcdd7287640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-15T00:11:54.677 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:11:54 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0[46684]: 2024-06-15T00:11:54.373+0000 7fcdd7287640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:00.046 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:11:59 smithi063 podman[79861]: 2024-06-15 00:11:59.776777817 +0000 UTC m=+5.509005392 container died 5ba4eb56bf192c464d40e25504320f93a4a7141ca07eba745bfd3db3def4733f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:00.427 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:00 smithi063 podman[79861]: 2024-06-15 00:12:00.249302957 +0000 UTC m=+5.981530572 container cleanup 5ba4eb56bf192c464d40e25504320f93a4a7141ca07eba745bfd3db3def4733f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:00.427 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:00 smithi063 bash[79861]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0 2024-06-15T00:12:00.880 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:00 smithi063 podman[79879]: 2024-06-15 00:12:00.550745804 +0000 UTC m=+0.763979853 container remove 5ba4eb56bf192c464d40e25504320f93a4a7141ca07eba745bfd3db3def4733f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-06-15T00:12:01.177 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:00 smithi063 podman[79967]: 2024-06-15 00:12:00.884747101 +0000 UTC m=+0.114397449 container create 1432796a2070f9374c36f08d282aa14b8157f4f7c3e41519aba813359e071c05 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240612, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2) 2024-06-15T00:12:01.177 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:00 smithi063 podman[79967]: 2024-06-15 00:12:00.788466998 +0000 UTC m=+0.018117357 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:01.177 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:01 smithi063 podman[79967]: 2024-06-15 00:12:01.112113925 +0000 UTC m=+0.341764280 container init 1432796a2070f9374c36f08d282aa14b8157f4f7c3e41519aba813359e071c05 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:12:01.178 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:01 smithi063 podman[79967]: 2024-06-15 00:12:01.116036606 +0000 UTC m=+0.345686954 container start 1432796a2070f9374c36f08d282aa14b8157f4f7c3e41519aba813359e071c05 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-15T00:12:01.652 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:01 smithi063 podman[79967]: 2024-06-15 00:12:01.203652754 +0000 UTC m=+0.433303100 container attach 1432796a2070f9374c36f08d282aa14b8157f4f7c3e41519aba813359e071c05 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:01.652 INFO:journalctl@ceph.osd.0.smithi063.stdout:Jun 15 00:12:01 smithi063 podman[79967]: 2024-06-15 00:12:01.372548466 +0000 UTC m=+0.602198812 container died 1432796a2070f9374c36f08d282aa14b8157f4f7c3e41519aba813359e071c05 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=quincy-e47ff4b) 2024-06-15T00:12:02.010 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-15T00:12:02.050 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:02.051 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-15T00:12:02.051 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-15T00:12:02.051 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.1 2024-06-15T00:12:02.382 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:02 smithi063 systemd[1]: Stopping Ceph osd.1 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:12:02.677 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:02 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1[53893]: 2024-06-15T00:12:02.384+0000 7f5ebeede640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:12:02.677 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:02 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1[53893]: 2024-06-15T00:12:02.384+0000 7f5ebeede640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-15T00:12:02.677 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:02 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1[53893]: 2024-06-15T00:12:02.384+0000 7f5ebeede640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:08.177 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:07 smithi063 podman[80071]: 2024-06-15 00:12:07.918890448 +0000 UTC m=+5.638956885 container died fe03aa7af30a379415963ffdb40127eb065ebc76cf186fb64ac95fe68bd0d699 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-15T00:12:08.656 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:08 smithi063 podman[80071]: 2024-06-15 00:12:08.393364987 +0000 UTC m=+6.113431461 container cleanup fe03aa7af30a379415963ffdb40127eb065ebc76cf186fb64ac95fe68bd0d699 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-06-15T00:12:08.657 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:08 smithi063 bash[80071]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1 2024-06-15T00:12:08.927 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:08 smithi063 podman[80088]: 2024-06-15 00:12:08.66111549 +0000 UTC m=+0.734451707 container remove fe03aa7af30a379415963ffdb40127eb065ebc76cf186fb64ac95fe68bd0d699 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-06-15T00:12:09.302 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:09.028771435 +0000 UTC m=+0.111676113 container create e9b0042384f7ffffe7a2f315ed9d7ac4d94a9cd2db585f789c2900f5ec5e8fd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-15T00:12:09.302 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:08.934717325 +0000 UTC m=+0.017622006 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:09.302 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:09.222956958 +0000 UTC m=+0.305861652 container init e9b0042384f7ffffe7a2f315ed9d7ac4d94a9cd2db585f789c2900f5ec5e8fd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-06-15T00:12:09.302 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:09.226334853 +0000 UTC m=+0.309239540 container start e9b0042384f7ffffe7a2f315ed9d7ac4d94a9cd2db585f789c2900f5ec5e8fd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-15T00:12:09.302 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:09.306101846 +0000 UTC m=+0.389006527 container attach e9b0042384f7ffffe7a2f315ed9d7ac4d94a9cd2db585f789c2900f5ec5e8fd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:12:09.677 INFO:journalctl@ceph.osd.1.smithi063.stdout:Jun 15 00:12:09 smithi063 podman[80179]: 2024-06-15 00:12:09.487942385 +0000 UTC m=+0.570847064 container died e9b0042384f7ffffe7a2f315ed9d7ac4d94a9cd2db585f789c2900f5ec5e8fd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-15T00:12:10.183 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-15T00:12:10.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:10.226 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-15T00:12:10.226 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-15T00:12:10.226 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.2 2024-06-15T00:12:10.427 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:10 smithi063 systemd[1]: Stopping Ceph osd.2 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:12:10.927 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:10 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2[61568]: 2024-06-15T00:12:10.604+0000 7f1e8dc9f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:12:10.927 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:10 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2[61568]: 2024-06-15T00:12:10.604+0000 7f1e8dc9f640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-15T00:12:10.927 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:10 smithi063 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2[61568]: 2024-06-15T00:12:10.604+0000 7f1e8dc9f640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:16.359 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:16 smithi063 podman[80282]: 2024-06-15 00:12:16.077437151 +0000 UTC m=+5.605862425 container died d913d1d8014556661021e1357ce6d685902e8604b1b67d8451b4fd611d9f55a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:12:16.677 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:16 smithi063 podman[80282]: 2024-06-15 00:12:16.56233836 +0000 UTC m=+6.090763640 container cleanup d913d1d8014556661021e1357ce6d685902e8604b1b67d8451b4fd611d9f55a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-15T00:12:16.677 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:16 smithi063 bash[80282]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2 2024-06-15T00:12:17.146 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:16 smithi063 podman[80312]: 2024-06-15 00:12:16.805766321 +0000 UTC m=+0.719224857 container remove d913d1d8014556661021e1357ce6d685902e8604b1b67d8451b4fd611d9f55a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-15T00:12:17.399 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.054218058 +0000 UTC m=+0.017935164 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:17.399 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.181227293 +0000 UTC m=+0.144944428 container create 02a67a247899c64a2f8fba53f407a7e98aa344101e5b6e3a51258ce587fb6a69 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240612, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-15T00:12:17.399 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.400386749 +0000 UTC m=+0.364103856 container init 02a67a247899c64a2f8fba53f407a7e98aa344101e5b6e3a51258ce587fb6a69 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2-deactivate, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-06-15T00:12:17.400 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.403699581 +0000 UTC m=+0.367416674 container start 02a67a247899c64a2f8fba53f407a7e98aa344101e5b6e3a51258ce587fb6a69 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.build-date=20240612, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-06-15T00:12:17.657 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.500309876 +0000 UTC m=+0.464026985 container attach 02a67a247899c64a2f8fba53f407a7e98aa344101e5b6e3a51258ce587fb6a69 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2-deactivate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-06-15T00:12:17.927 INFO:journalctl@ceph.osd.2.smithi063.stdout:Jun 15 00:12:17 smithi063 podman[80401]: 2024-06-15 00:12:17.661565408 +0000 UTC m=+0.625282500 container died 02a67a247899c64a2f8fba53f407a7e98aa344101e5b6e3a51258ce587fb6a69 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-15T00:12:18.235 DEBUG:teuthology.orchestra.run.smithi063:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-15T00:12:18.277 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:18.277 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-15T00:12:18.277 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-15T00:12:18.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.3 2024-06-15T00:12:18.582 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:18 smithi136 systemd[1]: Stopping Ceph osd.3 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:12:19.082 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:18 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:12:18.700+0000 7f62b100c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:12:19.083 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:18 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:12:18.700+0000 7f62b100c640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-15T00:12:19.083 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:18 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:12:18.700+0000 7f62b100c640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:21.332 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:20 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:20.844+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:22.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:21 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:21.805+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:23.082 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:22 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3[41339]: 2024-06-15T00:12:22.862+0000 7f62aa61e640 -1 osd.3 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:58.566888+0000 front 2024-06-15T00:11:58.567025+0000 (oldest deadline 2024-06-15T00:12:22.067044+0000) 2024-06-15T00:12:23.083 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:22 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:22.761+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:23.778 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:23 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:23.411+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:24.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:23 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:23.781+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:24.392 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:24 smithi136 podman[59539]: 2024-06-15 00:12:24.118481115 +0000 UTC m=+5.637272485 container died 7813a2421921f7a941e6ff2c9161686761565e19fbdf9e78c259738f87762e01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-06-15T00:12:24.755 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:24 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:24.395+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:24.755 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:24 smithi136 podman[59539]: 2024-06-15 00:12:24.516735552 +0000 UTC m=+6.035526922 container cleanup 7813a2421921f7a941e6ff2c9161686761565e19fbdf9e78c259738f87762e01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:12:24.755 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:24 smithi136 bash[59539]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3 2024-06-15T00:12:25.080 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:24 smithi136 podman[59558]: 2024-06-15 00:12:24.759440937 +0000 UTC m=+0.631873597 container remove 7813a2421921f7a941e6ff2c9161686761565e19fbdf9e78c259738f87762e01 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.build-date=20240612, ceph=True, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS) 2024-06-15T00:12:25.080 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:24 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:24.812+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:25.333 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:24.986835898 +0000 UTC m=+0.017600294 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:25.333 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:25.085136226 +0000 UTC m=+0.115900618 container create 304b98ad76a9abc0f520b906d6d25363d2287fa97b5fa044cede938c512c2f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0) 2024-06-15T00:12:25.333 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:25.321591084 +0000 UTC m=+0.352355476 container init 304b98ad76a9abc0f520b906d6d25363d2287fa97b5fa044cede938c512c2f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-15T00:12:25.333 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:25.325081823 +0000 UTC m=+0.355846216 container start 304b98ad76a9abc0f520b906d6d25363d2287fa97b5fa044cede938c512c2f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:12:25.785 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:25 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:25.362+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:25.785 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:25.438558855 +0000 UTC m=+0.469323251 container attach 304b98ad76a9abc0f520b906d6d25363d2287fa97b5fa044cede938c512c2f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-06-15T00:12:25.785 INFO:journalctl@ceph.osd.3.smithi136.stdout:Jun 15 00:12:25 smithi136 podman[59646]: 2024-06-15 00:12:25.567981441 +0000 UTC m=+0.598745834 container died 304b98ad76a9abc0f520b906d6d25363d2287fa97b5fa044cede938c512c2f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:26.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:25 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:25.787+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:26.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-15T00:12:26.252 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:26.252 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-15T00:12:26.252 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-15T00:12:26.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.4 2024-06-15T00:12:26.582 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:26 smithi136 systemd[1]: Stopping Ceph osd.4 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:12:26.583 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:26 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:26.348+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:26.832 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:26 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:26.585+0000 7fa4ce7ed640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:12:26.832 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:26 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:26.585+0000 7fa4ce7ed640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-15T00:12:26.832 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:26 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:26.585+0000 7fa4ce7ed640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:26.833 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:26 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:26.793+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:27.761 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:27 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:27.391+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:28.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:27 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:27.765+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:28.767 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:28 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:28.354+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:29.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:28 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:28.771+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:29.582 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:29 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:29.325+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:30.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:29 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:29.741+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:30.699 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:30 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:30.359+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:30.699 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:30 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:30.359+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:31.082 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:30 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4[47689]: 2024-06-15T00:12:30.702+0000 7fa4c9e03640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:55.264388+0000 front 2024-06-15T00:11:55.264516+0000 (oldest deadline 2024-06-15T00:12:19.964324+0000) 2024-06-15T00:12:31.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:31 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:31.386+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:31.833 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:31 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:31.386+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:32.332 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:32 smithi136 podman[59749]: 2024-06-15 00:12:32.042295256 +0000 UTC m=+5.565522808 container died 49ec9cb7a085dd4f9af560f10d5839782e01a167d9541dbe34163d439d0dde9d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-06-15T00:12:32.582 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:32 smithi136 podman[59749]: 2024-06-15 00:12:32.427037114 +0000 UTC m=+5.950264672 container cleanup 49ec9cb7a085dd4f9af560f10d5839782e01a167d9541dbe34163d439d0dde9d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-15T00:12:32.583 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:32 smithi136 bash[59749]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4 2024-06-15T00:12:32.583 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:32 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:32.404+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:32.583 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:32 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:32.404+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:33.041 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:32 smithi136 podman[59765]: 2024-06-15 00:12:32.694856124 +0000 UTC m=+0.650631752 container remove 49ec9cb7a085dd4f9af560f10d5839782e01a167d9541dbe34163d439d0dde9d (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0) 2024-06-15T00:12:33.308 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:33.045943866 +0000 UTC m=+0.111855174 container create 2658577a8c8bd9bc1322080fb7353b5423b221b4285adb34d9bb2ae94604687b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-06-15T00:12:33.308 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:32.951179725 +0000 UTC m=+0.017090992 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:33.582 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:33 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:33.442+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:33.582 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:33 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:33.442+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:33.583 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:33.30675272 +0000 UTC m=+0.372663997 container init 2658577a8c8bd9bc1322080fb7353b5423b221b4285adb34d9bb2ae94604687b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:33.583 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:33.310907838 +0000 UTC m=+0.376819111 container start 2658577a8c8bd9bc1322080fb7353b5423b221b4285adb34d9bb2ae94604687b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-06-15T00:12:33.583 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:33.406731195 +0000 UTC m=+0.472642470 container attach 2658577a8c8bd9bc1322080fb7353b5423b221b4285adb34d9bb2ae94604687b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-15T00:12:33.583 INFO:journalctl@ceph.osd.4.smithi136.stdout:Jun 15 00:12:33 smithi136 podman[59854]: 2024-06-15 00:12:33.556395794 +0000 UTC m=+0.622307063 container died 2658577a8c8bd9bc1322080fb7353b5423b221b4285adb34d9bb2ae94604687b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0) 2024-06-15T00:12:34.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-15T00:12:34.162 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:34.163 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-15T00:12:34.163 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-15T00:12:34.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.5 2024-06-15T00:12:34.408 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 systemd[1]: Stopping Ceph osd.5 for 25f32e20-2aaa-11ef-bc9d-c7b262605968... 2024-06-15T00:12:34.408 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:34.412+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:34.409 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:34.412+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:34.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:34.513+0000 7efd83338640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:12:34.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:34.513+0000 7efd83338640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-15T00:12:34.833 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:34 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:34.513+0000 7efd83338640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:12:35.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:35 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:35.401+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:35.833 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:35 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:35.401+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:36.583 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:36 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:36.369+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:36.583 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:36 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:36.369+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:37.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:37 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:37.398+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:37.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:37 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:37.398+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:38.834 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:38 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:38.424+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:38.834 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:38 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:38.424+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:39.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:39 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:39.402+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6806 osd.0 since back 2024-06-15T00:11:56.215946+0000 front 2024-06-15T00:11:56.216106+0000 (oldest deadline 2024-06-15T00:12:22.616360+0000) 2024-06-15T00:12:39.833 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:39 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:39.402+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6814 osd.1 since back 2024-06-15T00:12:04.317386+0000 front 2024-06-15T00:12:04.317424+0000 (oldest deadline 2024-06-15T00:12:30.217293+0000) 2024-06-15T00:12:39.833 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:39 smithi136 ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5[54544]: 2024-06-15T00:12:39.402+0000 7efd80952640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.63:6822 osd.2 since back 2024-06-15T00:12:15.518039+0000 front 2024-06-15T00:12:15.518050+0000 (oldest deadline 2024-06-15T00:12:39.017860+0000) 2024-06-15T00:12:40.277 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:39 smithi136 podman[59955]: 2024-06-15 00:12:39.978716009 +0000 UTC m=+5.594272548 container died a0241efcb4da4fe26da3314c44cdbf600f2a7bf27e4a527f002aa136f5f7cff5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:12:40.582 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:40 smithi136 podman[59955]: 2024-06-15 00:12:40.381525664 +0000 UTC m=+5.997082196 container cleanup a0241efcb4da4fe26da3314c44cdbf600f2a7bf27e4a527f002aa136f5f7cff5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:12:40.582 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:40 smithi136 bash[59955]: ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5 2024-06-15T00:12:41.046 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:40 smithi136 podman[59971]: 2024-06-15 00:12:40.607972566 +0000 UTC m=+0.620030797 container remove a0241efcb4da4fe26da3314c44cdbf600f2a7bf27e4a527f002aa136f5f7cff5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-06-15T00:12:41.332 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:41.050531102 +0000 UTC m=+0.108397873 container create ce176d81b40990aab30774946f9d067c92b48e27499c0e6f5f1e094fd6261bf6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2024-06-15T00:12:41.332 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:40.958863761 +0000 UTC m=+0.016730499 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:12:41.333 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:41.294648502 +0000 UTC m=+0.352515240 container init ce176d81b40990aab30774946f9d067c92b48e27499c0e6f5f1e094fd6261bf6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b) 2024-06-15T00:12:41.333 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:41.29782543 +0000 UTC m=+0.355692160 container start ce176d81b40990aab30774946f9d067c92b48e27499c0e6f5f1e094fd6261bf6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5-deactivate, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-15T00:12:41.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:41.386258899 +0000 UTC m=+0.444125628 container attach ce176d81b40990aab30774946f9d067c92b48e27499c0e6f5f1e094fd6261bf6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:12:41.832 INFO:journalctl@ceph.osd.5.smithi136.stdout:Jun 15 00:12:41 smithi136 podman[60062]: 2024-06-15 00:12:41.543694512 +0000 UTC m=+0.601561249 container died ce176d81b40990aab30774946f9d067c92b48e27499c0e6f5f1e094fd6261bf6 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968-osd-5-deactivate, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, ceph=True) 2024-06-15T00:12:42.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo pkill -f 'journalctl -f -n 0 -u ceph-25f32e20-2aaa-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-15T00:12:42.146 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:12:42.146 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-15T00:12:42.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-15T00:12:51.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-15T00:12:56.291 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:12:56.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:12:56.353 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-15T00:12:56.355 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi063/crash 2024-06-15T00:12:56.356 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash -- . 2024-06-15T00:12:56.407 INFO:teuthology.orchestra.run.smithi063.stderr:tar: /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-15T00:12:56.407 INFO:teuthology.orchestra.run.smithi063.stderr:tar: Error is not recoverable: exiting now 2024-06-15T00:12:56.409 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi136/crash 2024-06-15T00:12:56.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash -- . 2024-06-15T00:12:56.443 INFO:teuthology.orchestra.run.smithi136.stderr:tar: /var/lib/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-15T00:12:56.443 INFO:teuthology.orchestra.run.smithi136.stderr:tar: Error is not recoverable: exiting now 2024-06-15T00:12:56.444 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-15T00:12:56.444 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-06-15T00:12:56.480 INFO:teuthology.orchestra.run.smithi063.stderr:grep: /var/log/ceph/25f32e20-2aaa-11ef-bc9d-c7b262605968/ceph.log: No such file or directory 2024-06-15T00:12:56.482 INFO:tasks.cephadm:Compressing logs... 2024-06-15T00:12:56.482 DEBUG:teuthology.orchestra.run.smithi063:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:12:56.525 DEBUG:teuthology.orchestra.run.smithi136:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:12:56.548 INFO:teuthology.orchestra.run.smithi063.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-15T00:12:56.549 INFO:teuthology.orchestra.run.smithi136.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-15T00:12:57.298 INFO:tasks.cephadm:Archiving logs... 2024-06-15T00:12:57.298 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi063/log 2024-06-15T00:12:57.299 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T00:12:57.487 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi136/log 2024-06-15T00:12:57.488 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T00:12:57.556 INFO:tasks.cephadm:Removing cluster... 2024-06-15T00:12:57.556 DEBUG:teuthology.orchestra.run.smithi063:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 --force 2024-06-15T00:12:58.044 DEBUG:teuthology.orchestra.run.smithi136:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 25f32e20-2aaa-11ef-bc9d-c7b262605968 --force 2024-06-15T00:12:58.516 INFO:tasks.cephadm:Removing cephadm ... 2024-06-15T00:12:58.516 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-15T00:12:58.534 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-15T00:12:58.549 INFO:tasks.cephadm:Teardown complete 2024-06-15T00:12:58.550 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:12:58.739 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=40bb2ed2a76a4404b1dd7f46ad3be34b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:12:58.742 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-15T00:12:58.751 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-15T00:12:58.780 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-15T00:12:58.780 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T00:12:58.792 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T00:12:58.868 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-15T00:12:58.876 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-15T00:12:58.876 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:12:58.879 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:12:58.893 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:12:58.894 INFO:teuthology.orchestra.run.smithi063.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:12:58.897 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:12:58.897 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-06-15T00:12:58.898 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +1428us[+1441us] +/- 53ms 2024-06-15T00:12:58.898 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 -700us[ -700us] +/- 54ms 2024-06-15T00:12:58.898 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -1192us[-1192us] +/- 83ms 2024-06-15T00:12:58.898 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:12:58.899 INFO:teuthology.orchestra.run.smithi063.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:12:58.899 INFO:teuthology.orchestra.run.smithi063.stdout:=============================================================================== 2024-06-15T00:12:58.899 INFO:teuthology.orchestra.run.smithi063.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +1459us[+1471us] +/- 53ms 2024-06-15T00:12:58.899 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -666us[ -655us] +/- 54ms 2024-06-15T00:12:58.899 INFO:teuthology.orchestra.run.smithi063.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 24 -1142us[-1142us] +/- 83ms 2024-06-15T00:12:58.900 INFO:teuthology.orchestra.run.smithi063.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:12:58.900 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-15T00:12:58.909 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-15T00:12:58.909 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-15T00:12:58.930 DEBUG:teuthology.orchestra.run.smithi063:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-15T00:12:59.020 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-15T00:12:59.085 DEBUG:teuthology.orchestra.run.smithi063:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:12:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:12:59.117 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:12:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:12:59.144 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-15T00:12:59.153 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-15T00:12:59.172 INFO:teuthology.task.internal:Duration was 1201.699991 seconds 2024-06-15T00:12:59.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-15T00:12:59.204 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-15T00:12:59.204 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T00:12:59.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T00:12:59.245 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:12:59.248 INFO:teuthology.orchestra.run.smithi063.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:12:59.615 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-15T00:12:59.615 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2024-06-15T00:12:59.616 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T00:12:59.638 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-06-15T00:12:59.639 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T00:12:59.675 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-15T00:12:59.675 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:12:59.681 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:12:59.740 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-15T00:12:59.741 DEBUG:teuthology.orchestra.run.smithi063:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T00:12:59.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T00:13:00.135 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-15T00:13:00.145 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-15T00:13:00.146 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T00:13:00.171 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T00:13:00.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-15T00:13:00.212 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:13:00.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:13:00.236 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2024-06-15T00:13:00.260 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-06-15T00:13:00.283 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:13:00.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:13:00.322 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:13:00.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:13:00.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-15T00:13:00.347 INFO:teuthology.task.internal:Transferring archived files... 2024-06-15T00:13:00.348 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi063 2024-06-15T00:13:00.348 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T00:13:00.406 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756057/remote/smithi136 2024-06-15T00:13:00.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T00:13:00.441 INFO:teuthology.task.internal:Removing archive directory... 2024-06-15T00:13:00.441 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T00:13:00.444 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T00:13:00.490 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-15T00:13:00.505 INFO:teuthology.task.internal:Not uploading archives. 2024-06-15T00:13:00.506 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-15T00:13:00.531 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-15T00:13:00.531 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T00:13:00.534 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T00:13:00.547 INFO:teuthology.orchestra.run.smithi063.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 00:13 /home/ubuntu/cephtest 2024-06-15T00:13:00.548 INFO:teuthology.orchestra.run.smithi136.stdout: 266045 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 00:13 /home/ubuntu/cephtest 2024-06-15T00:13:00.549 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-15T00:13:00.566 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-15T00:13:00.692 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client/fuse 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1201.6999907493591 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=40bb2ed2a76a4404b1dd7f46ad3be34b status: fail success: false 2024-06-15T00:13:00.692 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T00:13:00.772 INFO:teuthology.run:FAIL