2024-06-06T02:58:07.106 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-06T02:58:07.107 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --owner scheduled_teuthology@teuthology --name teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004 -- /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/orig.config.yaml 2024-06-06T02:58:07.167 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T02:58:07.264 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004 branch: squid description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7743004' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba kclient: syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5050 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 416/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 3d746ca5346f34eef7e826a1eb71e59e02b682ae targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ7GfLphFIfQtM2QFSSUe3CXo1iZC7DYGHvpGvRVmDD1nXv54mb9pjXyPD9uwaP+Hau/XJdAIo4B5Fdnsboyk5M= smithi173.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMjTOH5pn4mBXzYyOMPlVqZ0GNS27MiWQTk2ktGjS/AkDK0DW9TlQ4UHlSEdfW6XOezdxzZtxY/Bk7QjJdFVrBI= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /home/teuthology/src/git.ceph.com_ceph_squid/qa/suites/fs/upgrade/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-05_21:24:16 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-06T02:58:07.264 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa; will attempt to use it 2024-06-06T02:58:07.265 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks 2024-06-06T02:58:07.265 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-06T02:58:07.267 INFO:teuthology.task.internal:Checking packages... 2024-06-06T02:58:07.294 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-06T02:58:07.294 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T02:58:07.294 INFO:teuthology.packaging:ref: None 2024-06-06T02:58:07.294 INFO:teuthology.packaging:tag: None 2024-06-06T02:58:07.294 INFO:teuthology.packaging:branch: squid 2024-06-06T02:58:07.294 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T02:58:07.295 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-06T02:58:07.612 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2960.g3d746ca5 2024-06-06T02:58:07.614 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-06T02:58:07.675 INFO:teuthology.task.internal:no buildpackages task found 2024-06-06T02:58:07.678 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-06T02:58:07.696 INFO:teuthology.task.internal:Saving configuration 2024-06-06T02:58:07.715 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-06T02:58:07.721 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-06T02:58:07.747 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 02:53:02.383910', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ7GfLphFIfQtM2QFSSUe3CXo1iZC7DYGHvpGvRVmDD1nXv54mb9pjXyPD9uwaP+Hau/XJdAIo4B5Fdnsboyk5M='} 2024-06-06T02:58:07.771 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi173.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-06 02:53:02.385627', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMjTOH5pn4mBXzYyOMPlVqZ0GNS27MiWQTk2ktGjS/AkDK0DW9TlQ4UHlSEdfW6XOezdxzZtxY/Bk7QjJdFVrBI='} 2024-06-06T02:58:07.771 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-06T02:58:07.778 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-06T02:58:07.778 INFO:teuthology.task.internal:roles: ubuntu@smithi173.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-06T02:58:07.778 INFO:teuthology.run_tasks:Running task console_log... 2024-06-06T02:58:07.853 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f35c2ddbbe0>, signals=[15]) 2024-06-06T02:58:07.853 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-06T02:58:07.860 INFO:teuthology.task.internal:Opening connections... 2024-06-06T02:58:07.860 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-06-06T02:58:07.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T02:58:07.941 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi173.front.sepia.ceph.com 2024-06-06T02:58:07.942 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T02:58:08.015 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-06T02:58:08.025 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-06-06T02:58:08.043 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-06-06T02:58:08.043 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="CentOS Stream" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="9" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:ID="centos" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE="rhel fedora" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="9" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:PLATFORM_ID="platform:el9" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T02:58:08.103 INFO:teuthology.orchestra.run.smithi153.stdout:ANSI_COLOR="0;31" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:LOGO="fedora-logo-icon" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://centos.org/" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T02:58:08.104 INFO:teuthology.orchestra.run.smithi153.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T02:58:08.105 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-06-06T02:58:08.122 DEBUG:teuthology.orchestra.run.smithi173:> uname -m 2024-06-06T02:58:08.139 INFO:teuthology.orchestra.run.smithi173.stdout:x86_64 2024-06-06T02:58:08.140 DEBUG:teuthology.orchestra.run.smithi173:> cat /etc/os-release 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:NAME="CentOS Stream" 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION="9" 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:ID="centos" 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:ID_LIKE="rhel fedora" 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:VERSION_ID="9" 2024-06-06T02:58:08.197 INFO:teuthology.orchestra.run.smithi173.stdout:PLATFORM_ID="platform:el9" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:ANSI_COLOR="0;31" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:LOGO="fedora-logo-icon" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:HOME_URL="https://centos.org/" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-06T02:58:08.198 INFO:teuthology.orchestra.run.smithi173.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-06T02:58:08.198 INFO:teuthology.lock.ops:Updating smithi173.front.sepia.ceph.com on lock server 2024-06-06T02:58:08.222 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-06T02:58:08.234 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-06T02:58:08.289 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-06T02:58:08.289 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-06-06T02:58:08.292 DEBUG:teuthology.orchestra.run.smithi173:> test '!' -e /home/ubuntu/cephtest 2024-06-06T02:58:08.308 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-06T02:58:08.319 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-06T02:58:08.319 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-06-06T02:58:08.349 DEBUG:teuthology.orchestra.run.smithi173:> test -z $(ls -A /var/lib/ceph) 2024-06-06T02:58:08.381 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-06T02:58:08.441 INFO:teuthology.run_tasks:Running task kernel... 2024-06-06T02:58:08.465 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-06T02:58:08.465 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-06T02:58:08.465 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T02:58:08.466 DEBUG:teuthology.orchestra.run.smithi173:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-06T02:58:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T02:58:08.481 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-06-06T02:58:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T02:58:08.482 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-06-06T02:58:08.537 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-437.el9.x86_64 2024-06-06T02:58:08.537 INFO:teuthology.task.kernel:Running kernel on smithi173: 5.14.0-437.el9.x86_64 2024-06-06T02:58:08.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum install -y kernel 2024-06-06T02:58:08.539 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-437.el9.x86_64 2024-06-06T02:58:08.539 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.14.0-437.el9.x86_64 2024-06-06T02:58:08.539 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum install -y kernel 2024-06-06T02:58:10.750 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.1 MB 00:00 2024-06-06T02:58:10.842 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-06T02:58:12.548 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-06-06T02:58:12.649 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 28 MB/s | 19 MB 00:00 2024-06-06T02:58:16.733 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-06-06T02:58:16.915 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.2 MB 00:00 2024-06-06T02:58:18.366 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 60 kB/s | 17 kB 00:00 2024-06-06T02:58:18.573 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 55 kB/s | 17 kB 00:00 2024-06-06T02:58:21.269 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 21 MB 00:02 2024-06-06T02:58:21.293 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 21 MB 00:02 2024-06-06T02:58:26.416 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-06-06T02:58:26.581 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-06-06T02:58:28.447 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T02:58:28.447 INFO:teuthology.orchestra.run.smithi173.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T02:58:28.541 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T02:58:28.548 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T02:58:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-06-06T02:58:28.548 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T02:58:28.548 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T02:58:28.548 INFO:teuthology.orchestra.run.smithi173.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-06-06T02:58:28.549 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:58:28.550 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 80 M 2024-06-06T02:58:28.550 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 124 M 2024-06-06T02:58:28.550 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T02:58:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-06T02:58:28.570 INFO:teuthology.orchestra.run.smithi153.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-06T02:58:28.663 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-06T02:58:28.674 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 80 M 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 124 M 2024-06-06T02:58:28.675 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T02:58:28.701 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.703 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.710 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.798 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.801 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.807 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T02:58:28.892 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.2 MB/s | 893 kB 00:00 2024-06-06T02:58:28.974 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.4 MB/s | 893 kB 00:00 2024-06-06T02:58:29.201 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-06-06T02:58:29.233 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-06-06T02:58:29.651 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 34 MB/s | 35 MB 00:01 2024-06-06T02:58:29.658 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 37 MB/s | 35 MB 00:00 2024-06-06T02:58:29.975 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 29 MB/s | 29 MB 00:01 2024-06-06T02:58:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T02:58:29.978 INFO:teuthology.orchestra.run.smithi153.stdout:Total 62 MB/s | 80 MB 00:01 2024-06-06T02:58:30.068 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 25 MB/s | 29 MB 00:01 2024-06-06T02:58:30.069 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T02:58:30.070 INFO:teuthology.orchestra.run.smithi173.stdout:Total 53 MB/s | 80 MB 00:01 2024-06-06T02:58:30.675 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T02:58:30.770 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T02:58:31.029 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T02:58:31.030 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T02:58:31.122 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T02:58:31.122 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T02:58:31.982 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T02:58:31.982 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T02:58:32.082 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T02:58:32.082 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T02:58:34.403 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T02:58:34.486 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T02:58:35.197 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T02:58:35.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-06T02:58:35.284 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:58:35.358 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:58:37.928 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:58:38.050 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:58:38.077 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:58:38.200 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:58:42.841 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:58:42.888 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:58:42.891 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:58:43.005 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:58:47.643 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:58:47.726 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.044 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.084 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.141 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.180 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.451 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T02:59:17.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:59:17.451 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:59:17.493 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:17.493 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-06T02:59:17.493 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-06T02:59:17.493 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.262 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.263 INFO:teuthology.orchestra.run.smithi173.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.263 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:59:18.263 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.274 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.275 INFO:teuthology.orchestra.run.smithi153.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:59:18.275 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T02:59:18.622 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T02:59:18.645 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T02:59:18.690 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.692 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.692 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.692 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T02:59:18.692 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T02:59:18.692 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-06T02:59:18.692 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T02:59:18.692 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T02:59:18.723 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.724 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.724 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.724 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-06T02:59:18.724 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-06T02:59:18.724 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-06T02:59:18.724 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi173.front.sepia.ceph.com, path=None, version=distro) 2024-06-06T02:59:18.725 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q kernel | sort -rV | head -n 1 2024-06-06T02:59:18.745 INFO:teuthology.orchestra.run.smithi153.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.745 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.745 DEBUG:teuthology.orchestra.run.smithi153:> sudo rpm -qi grub2-tools 2024-06-06T02:59:18.776 INFO:teuthology.orchestra.run.smithi173.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-06T02:59:18.778 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.778 DEBUG:teuthology.orchestra.run.smithi173:> sudo rpm -qi grub2-tools 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Name : grub2-tools 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Epoch : 1 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Version : 2.06 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Release : 68.el9 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: x86_64 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Group : Unspecified 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Size : 8264254 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:License : GPLv3+ 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T02:59:18.782 INFO:teuthology.orchestra.run.smithi153.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Packager : builder@centos.org 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Vendor : CentOS 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Summary : Support tools for GRUB. 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:Description : 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout:hardware devices. 2024-06-06T02:59:18.783 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T02:59:18.784 INFO:teuthology.orchestra.run.smithi153.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T02:59:18.785 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T02:59:18.785 INFO:teuthology.task.kernel:Updating grub on smithi153 to boot 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.785 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T02:59:18.808 INFO:teuthology.orchestra.run.smithi173.stdout:Name : grub2-tools 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Epoch : 1 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Version : 2.06 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Release : 68.el9 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Architecture: x86_64 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Group : Unspecified 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Size : 8264254 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:License : GPLv3+ 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-06T02:59:18.809 INFO:teuthology.orchestra.run.smithi173.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:Packager : builder@centos.org 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:Vendor : CentOS 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:Summary : Support tools for GRUB. 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:Description : 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:hardware devices. 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T02:59:18.810 INFO:teuthology.orchestra.run.smithi173.stdout:This subpackage provides tools for support of all platforms. 2024-06-06T02:59:18.811 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-06T02:59:18.812 INFO:teuthology.task.kernel:Updating grub on smithi173 to boot 5.14.0-452.el9.x86_64 2024-06-06T02:59:18.812 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-06T02:59:19.352 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-06-06T02:59:19.398 INFO:teuthology.orchestra.run.smithi173.stderr:Generating grub configuration file ... 2024-06-06T02:59:20.783 INFO:teuthology.orchestra.run.smithi153.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T02:59:20.808 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-06-06T02:59:20.811 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-06-06T02:59:20.826 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.NKBAmPINef 2024-06-06T02:59:20.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.NKBAmPINef 2024-06-06T02:59:20.831 INFO:teuthology.orchestra.run.smithi173.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-06T02:59:20.855 INFO:teuthology.orchestra.run.smithi173.stderr:done 2024-06-06T02:59:20.858 DEBUG:teuthology.orchestra.run.smithi173:> mktemp 2024-06-06T02:59:20.873 INFO:teuthology.orchestra.run.smithi173.stdout:/tmp/tmp.Xn3BTDhdKs 2024-06-06T02:59:20.873 DEBUG:teuthology.orchestra.run.smithi173:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Xn3BTDhdKs 2024-06-06T02:59:20.892 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.NKBAmPINef 2024-06-06T02:59:20.939 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0666 /tmp/tmp.Xn3BTDhdKs 2024-06-06T02:59:21.027 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.NKBAmPINef is 6KB 2024-06-06T02:59:21.076 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.NKBAmPINef 2024-06-06T02:59:21.090 DEBUG:teuthology.orchestra.run.smithi153:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T02:59:21.092 DEBUG:teuthology.orchestra.remote:smithi173:/tmp/tmp.Xn3BTDhdKs is 6KB 2024-06-06T02:59:21.140 DEBUG:teuthology.orchestra.run.smithi173:> rm -fr /tmp/tmp.Xn3BTDhdKs 2024-06-06T02:59:21.153 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T02:59:21.153 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T02:59:21.153 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T02:59:21.153 INFO:teuthology.orchestra.run.smithi153.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T02:59:21.154 DEBUG:teuthology.orchestra.run.smithi153:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T02:59:21.155 DEBUG:teuthology.orchestra.run.smithi173:> sudo /bin/ls /boot/loader/entries || true 2024-06-06T02:59:21.223 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-06T02:59:21.223 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-06T02:59:21.223 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-06T02:59:21.224 INFO:teuthology.orchestra.run.smithi173.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-06T02:59:21.225 DEBUG:teuthology.orchestra.run.smithi173:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-06T02:59:21.305 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-06-06T02:59:21.375 DEBUG:teuthology.orchestra.run.smithi173:> sudo shutdown -r now 2024-06-06T02:59:51.334 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-06T02:59:51.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T02:59:51.336 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T02:59:51.378 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-06T02:59:51.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T02:59:51.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:00:09.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-06T03:00:09.890 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-06-06T03:00:18.898 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:00:18.899 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:00:18.901 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:00:18.901 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:00:21.954 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-06T03:00:21.986 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.173 2024-06-06T03:00:33.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:00:33.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:00:33.987 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:00:33.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:00:52.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-06-06T03:01:07.396 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:01:07.397 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:01:07.837 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-06-06T03:01:08.288 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:01:08.289 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-452.el9.x86_64"... 2024-06-06T03:01:08.289 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-06-06T03:01:08.304 INFO:teuthology.orchestra.run.smithi153.stdout:5.14.0-452.el9.x86_64 2024-06-06T03:01:08.304 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T03:01:08.304 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T03:01:08.304 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T03:01:09.305 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-06T03:01:09.305 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T03:01:09.371 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-06-06T03:01:09.406 DEBUG:teuthology.parallel:result is None 2024-06-06T03:01:33.989 DEBUG:teuthology.orchestra.remote:timed out 2024-06-06T03:01:48.990 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:01:48.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:01:49.333 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-06-06T03:01:49.697 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:01:49.697 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-452.el9.x86_64"... 2024-06-06T03:01:49.698 DEBUG:teuthology.orchestra.run.smithi173:> uname -r 2024-06-06T03:01:49.713 INFO:teuthology.orchestra.run.smithi173.stdout:5.14.0-452.el9.x86_64 2024-06-06T03:01:49.713 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-06T03:01:49.713 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-06T03:01:49.713 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-06T03:01:50.714 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-06T03:01:50.715 DEBUG:teuthology.orchestra.run.smithi173:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-06T03:01:50.793 INFO:teuthology.orchestra.run.smithi173.stdout:ttyS1 2024-06-06T03:01:50.834 DEBUG:teuthology.parallel:result is None 2024-06-06T03:01:50.834 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-06T03:01:50.865 INFO:teuthology.task.internal:Creating test directory... 2024-06-06T03:01:50.866 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T03:01:50.869 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-06T03:01:50.887 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-06T03:01:50.933 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-06T03:01:50.940 INFO:teuthology.task.internal:Creating archive directory... 2024-06-06T03:01:50.941 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T03:01:50.943 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-06T03:01:51.019 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-06T03:01:51.026 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-06T03:01:51.026 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T03:01:51.029 DEBUG:teuthology.orchestra.run.smithi173:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-06T03:01:51.065 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T03:01:51.066 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T03:01:51.076 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T03:01:51.078 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-06T03:01:51.080 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-06T03:01:51.087 INFO:teuthology.task.internal:Configuring sudo... 2024-06-06T03:01:51.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T03:01:51.123 DEBUG:teuthology.orchestra.run.smithi173:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-06T03:01:51.151 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-06T03:01:51.160 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-06T03:01:51.160 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T03:01:51.192 DEBUG:teuthology.orchestra.run.smithi173:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-06T03:01:51.208 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T03:01:51.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T03:01:51.342 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T03:01:51.424 DEBUG:teuthology.orchestra.run.smithi153:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T03:01:51.466 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:01:51.466 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T03:01:51.532 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T03:01:51.576 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-06T03:01:51.632 DEBUG:teuthology.orchestra.run.smithi173:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T03:01:51.715 DEBUG:teuthology.orchestra.run.smithi173:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-06T03:01:51.757 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:01:51.757 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-06T03:01:51.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-06-06T03:01:51.826 DEBUG:teuthology.orchestra.run.smithi173:> sudo service rsyslog restart 2024-06-06T03:01:51.869 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T03:01:51.904 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T03:01:52.289 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-06T03:01:52.305 INFO:teuthology.task.internal:Starting timer... 2024-06-06T03:01:52.305 INFO:teuthology.run_tasks:Running task pcp... 2024-06-06T03:01:52.321 INFO:teuthology.run_tasks:Running task selinux... 2024-06-06T03:01:52.330 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-06-06T03:01:52.331 DEBUG:teuthology.orchestra.run.smithi153:> sudo service auditd rotate 2024-06-06T03:01:52.385 INFO:teuthology.orchestra.run.smithi153.stdout:Rotating logs: 2024-06-06T03:01:52.387 DEBUG:teuthology.orchestra.run.smithi173:> sudo service auditd rotate 2024-06-06T03:01:52.435 INFO:teuthology.orchestra.run.smithi173.stdout:Rotating logs: 2024-06-06T03:01:52.437 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-06T03:01:52.437 DEBUG:teuthology.orchestra.run.smithi153:> /usr/sbin/getenforce 2024-06-06T03:01:52.474 INFO:teuthology.orchestra.run.smithi153.stdout:Permissive 2024-06-06T03:01:52.474 DEBUG:teuthology.orchestra.run.smithi173:> /usr/sbin/getenforce 2024-06-06T03:01:52.493 INFO:teuthology.orchestra.run.smithi173.stdout:Permissive 2024-06-06T03:01:52.493 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi153.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi173.front.sepia.ceph.com': 'permissive'} 2024-06-06T03:01:52.494 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T03:01:52.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:01:52.542 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T03:01:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:01:52.639 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-06T03:01:52.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo /usr/sbin/setenforce permissive 2024-06-06T03:01:52.687 DEBUG:teuthology.orchestra.run.smithi173:> sudo /usr/sbin/setenforce permissive 2024-06-06T03:01:52.713 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-06T03:01:52.723 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-06T03:01:52.734 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-06T03:01:52.736 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi173.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-06T03:06:10.829 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi173.front.sepia.ceph.com')] 2024-06-06T03:06:10.830 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:06:10.831 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:06:10.915 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-06-06T03:06:10.998 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-06-06T03:06:10.999 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:06:10.999 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi173.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-06T03:06:11.074 DEBUG:teuthology.orchestra.run.smithi173:> true 2024-06-06T03:06:11.150 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi173.front.sepia.ceph.com' 2024-06-06T03:06:11.150 INFO:teuthology.run_tasks:Running task clock... 2024-06-06T03:06:11.559 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-06T03:06:11.559 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T03:06:11.560 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T03:06:11.562 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-06T03:06:11.563 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T03:06:11.593 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T03:06:11.594 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-06T03:06:11.608 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T03:06:11.609 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-06T03:06:11.633 INFO:teuthology.orchestra.run.smithi173.stderr:sudo: ntpd: command not found 2024-06-06T03:06:11.636 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: ntpd: command not found 2024-06-06T03:06:11.645 INFO:teuthology.orchestra.run.smithi173.stdout:506 Cannot talk to daemon 2024-06-06T03:06:11.647 INFO:teuthology.orchestra.run.smithi153.stdout:506 Cannot talk to daemon 2024-06-06T03:06:11.659 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T03:06:11.662 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-06T03:06:11.673 INFO:teuthology.orchestra.run.smithi173.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T03:06:11.677 INFO:teuthology.orchestra.run.smithi153.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-06T03:06:11.727 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-06-06T03:06:11.730 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T03:06:11.730 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-06-06T03:06:11.730 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.731 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.731 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.731 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.742 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-06T03:06:11.745 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T03:06:11.745 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-06T03:06:11.745 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.745 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.745 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.746 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:06:11.746 INFO:teuthology.run_tasks:Running task install... 2024-06-06T03:06:11.755 DEBUG:teuthology.task.install:project ceph 2024-06-06T03:06:11.755 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-06T03:06:11.755 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T03:06:11.755 INFO:teuthology.task.install:Using flavor: default 2024-06-06T03:06:11.761 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-06T03:06:11.761 INFO:teuthology.task.install:extra packages: [] 2024-06-06T03:06:11.761 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T03:06:11.762 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:06:11.762 INFO:teuthology.packaging:ref: None 2024-06-06T03:06:11.762 INFO:teuthology.packaging:tag: None 2024-06-06T03:06:11.762 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:06:11.762 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:06:11.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T03:06:11.764 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-06T03:06:11.764 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:06:11.764 INFO:teuthology.packaging:ref: None 2024-06-06T03:06:11.764 INFO:teuthology.packaging:tag: None 2024-06-06T03:06:11.764 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:06:11.764 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:06:11.765 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T03:06:11.963 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/ 2024-06-06T03:06:11.963 INFO:teuthology.task.install.rpm:Package version is 17.2.7-992.g6ee68927 2024-06-06T03:06:11.996 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/ 2024-06-06T03:06:11.996 INFO:teuthology.task.install.rpm:Package version is 17.2.7-992.g6ee68927 2024-06-06T03:06:12.080 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T03:06:12.081 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:06:12.081 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T03:06:12.108 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T03:06:12.108 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:06:12.108 INFO:teuthology.packaging:ref: None 2024-06-06T03:06:12.108 INFO:teuthology.packaging:tag: None 2024-06-06T03:06:12.108 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:06:12.108 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:06:12.109 DEBUG:teuthology.orchestra.run.smithi153:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T03:06:12.169 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/quincy/6ee689278780be842e5800190ee2e2bd3d5027f6/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-06T03:06:12.169 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:06:12.169 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-06T03:06:12.175 DEBUG:teuthology.orchestra.run.smithi153:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T03:06:12.196 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-06T03:06:12.196 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:06:12.196 INFO:teuthology.packaging:ref: None 2024-06-06T03:06:12.196 INFO:teuthology.packaging:tag: None 2024-06-06T03:06:12.196 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:06:12.196 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:06:12.196 DEBUG:teuthology.orchestra.run.smithi173:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-06T03:06:12.251 DEBUG:teuthology.orchestra.run.smithi153:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T03:06:12.262 DEBUG:teuthology.orchestra.run.smithi173:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-06T03:06:12.315 INFO:teuthology.orchestra.run.smithi153.stdout:check_obsoletes = 1 2024-06-06T03:06:12.317 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum clean all 2024-06-06T03:06:12.337 DEBUG:teuthology.orchestra.run.smithi173:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-06T03:06:12.401 INFO:teuthology.orchestra.run.smithi173.stdout:check_obsoletes = 1 2024-06-06T03:06:12.403 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum clean all 2024-06-06T03:06:12.552 INFO:teuthology.orchestra.run.smithi153.stdout:45 files removed 2024-06-06T03:06:12.579 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-radosgw 2024-06-06T03:06:12.628 INFO:teuthology.orchestra.run.smithi173.stdout:45 files removed 2024-06-06T03:06:12.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-radosgw 2024-06-06T03:06:13.367 INFO:teuthology.orchestra.run.smithi173.stdout:ceph packages for x86_64 179 kB/s | 77 kB 00:00 2024-06-06T03:06:13.377 INFO:teuthology.orchestra.run.smithi153.stdout:ceph packages for x86_64 149 kB/s | 77 kB 00:00 2024-06-06T03:06:13.728 INFO:teuthology.orchestra.run.smithi153.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-06-06T03:06:13.800 INFO:teuthology.orchestra.run.smithi173.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-06T03:06:14.270 INFO:teuthology.orchestra.run.smithi153.stdout:ceph source packages 3.6 kB/s | 1.8 kB 00:00 2024-06-06T03:06:14.325 INFO:teuthology.orchestra.run.smithi173.stdout:ceph source packages 3.7 kB/s | 1.8 kB 00:00 2024-06-06T03:06:15.039 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-06T03:06:15.177 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.1 MB 00:00 2024-06-06T03:06:16.907 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-06-06T03:06:17.035 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-06-06T03:06:21.221 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-06-06T03:06:21.266 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.2 MB 00:00 2024-06-06T03:06:22.841 INFO:teuthology.orchestra.run.smithi153.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 17 kB 00:00 2024-06-06T03:06:22.942 INFO:teuthology.orchestra.run.smithi173.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 17 kB 00:00 2024-06-06T03:06:26.211 INFO:teuthology.orchestra.run.smithi153.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-06-06T03:06:26.274 INFO:teuthology.orchestra.run.smithi173.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 21 MB 00:03 2024-06-06T03:06:31.350 INFO:teuthology.orchestra.run.smithi153.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-06-06T03:06:31.429 INFO:teuthology.orchestra.run.smithi173.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-06-06T03:06:33.749 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:06:33.760 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:06:33.760 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 14 M 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrading: 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: librados2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.1 M 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base x86_64 2:17.2.7-992.g6ee68927.el9 ceph 5.0 M 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 22 M 2024-06-06T03:06:33.761 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 637 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 458 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 4.9 M 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 45 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 117 k 2024-06-06T03:06:33.762 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs x86_64 2:17.2.7-992.g6ee68927.el9 ceph 156 k 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados x86_64 2:17.2.7-992.g6ee68927.el9 ceph 321 k 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 292 k 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 99 k 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout:Install 22 Packages 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout:Upgrade 2 Packages 2024-06-06T03:06:33.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:06:33.764 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 58 M 2024-06-06T03:06:33.764 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:06:33.861 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:06:33.873 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:06:33.873 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:06:33.873 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 14 M 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrading: 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: librados2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.1 M 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base x86_64 2:17.2.7-992.g6ee68927.el9 ceph 5.0 M 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 22 M 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 637 k 2024-06-06T03:06:33.874 INFO:teuthology.orchestra.run.smithi173.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 458 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2 x86_64 2:17.2.7-992.g6ee68927.el9 ceph 4.9 M 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 45 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common x86_64 2:17.2.7-992.g6ee68927.el9 ceph 117 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs x86_64 2:17.2.7-992.g6ee68927.el9 ceph 156 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados x86_64 2:17.2.7-992.g6ee68927.el9 ceph 321 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 292 k 2024-06-06T03:06:33.875 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw x86_64 2:17.2.7-992.g6ee68927.el9 ceph 99 k 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Install 22 Packages 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Upgrade 2 Packages 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 58 M 2024-06-06T03:06:33.876 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:06:37.180 INFO:teuthology.orchestra.run.smithi173.stdout:(1/24): ceph-base-17.2.7-992.g6ee68927.el9.x86_ 1.6 MB/s | 5.0 MB 00:03 2024-06-06T03:06:37.964 INFO:teuthology.orchestra.run.smithi173.stdout:(2/24): ceph-selinux-17.2.7-992.g6ee68927.el9.x 32 kB/s | 25 kB 00:00 2024-06-06T03:06:38.358 INFO:teuthology.orchestra.run.smithi153.stdout:(1/24): ceph-base-17.2.7-992.g6ee68927.el9.x86_ 1.1 MB/s | 5.0 MB 00:04 2024-06-06T03:06:38.475 INFO:teuthology.orchestra.run.smithi153.stdout:(2/24): ceph-selinux-17.2.7-992.g6ee68927.el9.x 216 kB/s | 25 kB 00:00 2024-06-06T03:06:38.481 INFO:teuthology.orchestra.run.smithi173.stdout:(3/24): libcephfs2-17.2.7-992.g6ee68927.el9.x86 1.2 MB/s | 637 kB 00:00 2024-06-06T03:06:38.723 INFO:teuthology.orchestra.run.smithi173.stdout:(4/24): libradosstriper1-17.2.7-992.g6ee68927.e 1.9 MB/s | 458 kB 00:00 2024-06-06T03:06:38.901 INFO:teuthology.orchestra.run.smithi153.stdout:(3/24): libcephfs2-17.2.7-992.g6ee68927.el9.x86 1.5 MB/s | 637 kB 00:00 2024-06-06T03:06:39.276 INFO:teuthology.orchestra.run.smithi153.stdout:(4/24): libradosstriper1-17.2.7-992.g6ee68927.e 1.2 MB/s | 458 kB 00:00 2024-06-06T03:06:39.448 INFO:teuthology.orchestra.run.smithi173.stdout:(5/24): ceph-common-17.2.7-992.g6ee68927.el9.x8 4.2 MB/s | 22 MB 00:05 2024-06-06T03:06:39.590 INFO:teuthology.orchestra.run.smithi173.stdout:(6/24): python3-ceph-argparse-17.2.7-992.g6ee68 319 kB/s | 45 kB 00:00 2024-06-06T03:06:39.707 INFO:teuthology.orchestra.run.smithi173.stdout:(7/24): python3-ceph-common-17.2.7-992.g6ee6892 1.0 MB/s | 117 kB 00:00 2024-06-06T03:06:39.807 INFO:teuthology.orchestra.run.smithi173.stdout:(8/24): python3-cephfs-17.2.7-992.g6ee68927.el9 1.5 MB/s | 156 kB 00:00 2024-06-06T03:06:39.891 INFO:teuthology.orchestra.run.smithi173.stdout:(9/24): librgw2-17.2.7-992.g6ee68927.el9.x86_64 4.2 MB/s | 4.9 MB 00:01 2024-06-06T03:06:39.974 INFO:teuthology.orchestra.run.smithi173.stdout:(10/24): python3-rados-17.2.7-992.g6ee68927.el9 1.9 MB/s | 321 kB 00:00 2024-06-06T03:06:40.076 INFO:teuthology.orchestra.run.smithi173.stdout:(11/24): python3-rbd-17.2.7-992.g6ee68927.el9.x 1.5 MB/s | 292 kB 00:00 2024-06-06T03:06:40.100 INFO:teuthology.orchestra.run.smithi173.stdout:(12/24): python3-rgw-17.2.7-992.g6ee68927.el9.x 792 kB/s | 99 kB 00:00 2024-06-06T03:06:40.162 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:40.172 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:40.193 INFO:teuthology.orchestra.run.smithi153.stdout:(5/24): ceph-common-17.2.7-992.g6ee68927.el9.x8 3.6 MB/s | 22 MB 00:06 2024-06-06T03:06:40.268 INFO:teuthology.orchestra.run.smithi153.stdout:(6/24): python3-ceph-argparse-17.2.7-992.g6ee68 605 kB/s | 45 kB 00:00 2024-06-06T03:06:40.275 INFO:teuthology.orchestra.run.smithi173.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 259 kB/s | 45 kB 00:00 2024-06-06T03:06:40.278 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:40.317 INFO:teuthology.orchestra.run.smithi173.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 667 kB/s | 160 kB 00:00 2024-06-06T03:06:40.359 INFO:teuthology.orchestra.run.smithi153.stdout:(7/24): python3-ceph-common-17.2.7-992.g6ee6892 1.3 MB/s | 117 kB 00:00 2024-06-06T03:06:40.384 INFO:teuthology.orchestra.run.smithi173.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-06-06T03:06:40.418 INFO:teuthology.orchestra.run.smithi173.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 662 kB 00:00 2024-06-06T03:06:40.426 INFO:teuthology.orchestra.run.smithi153.stdout:(8/24): python3-cephfs-17.2.7-992.g6ee68927.el9 2.3 MB/s | 156 kB 00:00 2024-06-06T03:06:40.443 INFO:teuthology.orchestra.run.smithi173.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 722 kB/s | 42 kB 00:00 2024-06-06T03:06:40.476 INFO:teuthology.orchestra.run.smithi173.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-06T03:06:40.502 INFO:teuthology.orchestra.run.smithi173.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-06-06T03:06:40.527 INFO:teuthology.orchestra.run.smithi173.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-06T03:06:40.568 INFO:teuthology.orchestra.run.smithi153.stdout:(9/24): python3-rados-17.2.7-992.g6ee68927.el9. 2.2 MB/s | 321 kB 00:00 2024-06-06T03:06:40.702 INFO:teuthology.orchestra.run.smithi153.stdout:(10/24): python3-rbd-17.2.7-992.g6ee68927.el9.x 2.1 MB/s | 292 kB 00:00 2024-06-06T03:06:40.752 INFO:teuthology.orchestra.run.smithi173.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 6.3 MB/s | 1.6 MB 00:00 2024-06-06T03:06:40.769 INFO:teuthology.orchestra.run.smithi153.stdout:(11/24): python3-rgw-17.2.7-992.g6ee68927.el9.x 1.5 MB/s | 99 kB 00:00 2024-06-06T03:06:40.816 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] libpmemobj-1.12.1-1.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/libpmemobj-1.12.1-1.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:40.944 INFO:teuthology.orchestra.run.smithi153.stdout:(12/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 916 kB/s | 160 kB 00:00 2024-06-06T03:06:40.948 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] librabbitmq-0.11.0-7.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librabbitmq-0.11.0-7.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:41.011 INFO:teuthology.orchestra.run.smithi153.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 678 kB/s | 45 kB 00:00 2024-06-06T03:06:41.015 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] librdkafka-1.6.1-102.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/librdkafka-1.6.1-102.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:06:41.103 INFO:teuthology.orchestra.run.smithi153.stdout:(14/24): librdkafka-1.6.1-102.el9.x86_64.rpm 7.1 MB/s | 662 kB 00:00 2024-06-06T03:06:41.128 INFO:teuthology.orchestra.run.smithi173.stdout:(22/24): librados2-17.2.7-992.g6ee68927.el9.x86 5.1 MB/s | 3.1 MB 00:00 2024-06-06T03:06:41.154 INFO:teuthology.orchestra.run.smithi153.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-06-06T03:06:41.196 INFO:teuthology.orchestra.run.smithi153.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-06-06T03:06:41.211 INFO:teuthology.orchestra.run.smithi173.stdout:(23/24): librbd1-17.2.7-992.g6ee68927.el9.x86_6 6.4 MB/s | 2.9 MB 00:00 2024-06-06T03:06:41.263 INFO:teuthology.orchestra.run.smithi153.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-06-06T03:06:41.296 INFO:teuthology.orchestra.run.smithi153.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-06T03:06:41.330 INFO:teuthology.orchestra.run.smithi153.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-06-06T03:06:41.455 INFO:teuthology.orchestra.run.smithi153.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-06-06T03:06:41.989 INFO:teuthology.orchestra.run.smithi153.stdout:(21/24): librgw2-17.2.7-992.g6ee68927.el9.x86_6 1.8 MB/s | 4.9 MB 00:02 2024-06-06T03:06:42.039 INFO:teuthology.orchestra.run.smithi153.stdout:(22/24): librados2-17.2.7-992.g6ee68927.el9.x86 5.3 MB/s | 3.1 MB 00:00 2024-06-06T03:06:42.873 INFO:teuthology.orchestra.run.smithi153.stdout:(23/24): ceph-radosgw-17.2.7-992.g6ee68927.el9. 1.6 MB/s | 14 MB 00:08 2024-06-06T03:06:42.878 INFO:teuthology.orchestra.run.smithi173.stdout:(24/24): ceph-radosgw-17.2.7-992.g6ee68927.el9. 1.6 MB/s | 14 MB 00:08 2024-06-06T03:06:42.883 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:06:42.884 INFO:teuthology.orchestra.run.smithi173.stdout:Total 6.4 MB/s | 58 MB 00:09 2024-06-06T03:06:42.948 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:06:42.982 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:06:42.982 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:06:43.331 INFO:teuthology.orchestra.run.smithi153.stdout:(24/24): librbd1-17.2.7-992.g6ee68927.el9.x86_6 2.2 MB/s | 2.9 MB 00:01 2024-06-06T03:06:43.336 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:06:43.336 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.0 MB/s | 58 MB 00:09 2024-06-06T03:06:43.400 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:06:43.434 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:06:43.434 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:06:43.509 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:06:43.510 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:06:43.967 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:06:43.967 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:06:44.703 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:06:45.025 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-06T03:06:45.121 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:06:45.170 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:06:45.218 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:06:45.300 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T03:06:45.332 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:06:45.427 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:06:45.474 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-06T03:06:45.503 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-06T03:06:45.550 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-06T03:06:45.608 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:06:45.689 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:06:45.779 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T03:06:45.801 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:06:45.900 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:06:45.939 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-06T03:06:45.944 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-06T03:06:45.973 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-06T03:06:45.976 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T03:06:46.039 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T03:06:46.067 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-06T03:06:46.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 10/26 2024-06-06T03:06:46.139 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T03:06:46.208 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T03:06:46.237 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:06:46.303 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:06:46.343 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-06T03:06:46.387 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-06T03:06:46.409 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-06T03:06:46.420 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-06T03:06:46.446 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T03:06:46.509 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 8/26 2024-06-06T03:06:46.537 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-06T03:06:46.581 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 10/26 2024-06-06T03:06:46.625 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T03:06:46.695 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T03:06:46.715 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-06T03:06:46.724 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:06:46.758 INFO:teuthology.orchestra.run.smithi173.stdout: Upgrading : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T03:06:46.789 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:06:46.829 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-06T03:06:46.831 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T03:06:46.873 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-06T03:06:46.905 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 19/26 2024-06-06T03:06:46.906 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-06T03:06:46.935 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 20/26 2024-06-06T03:06:47.204 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-06T03:06:47.244 INFO:teuthology.orchestra.run.smithi153.stdout: Upgrading : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T03:06:47.334 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 18/26 2024-06-06T03:06:47.408 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 19/26 2024-06-06T03:06:47.439 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 20/26 2024-06-06T03:06:49.335 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:49.366 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:49.894 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:49.913 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:49.928 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:49.953 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T03:06:50.039 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T03:06:50.039 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T03:06:50.039 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T03:06:50.040 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:06:50.070 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:06:50.482 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 21/26 2024-06-06T03:06:50.524 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T03:06:50.625 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 22/26 2024-06-06T03:06:50.625 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-06T03:06:50.626 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-06T03:06:50.626 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:06:50.649 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:06:59.540 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:06:59.540 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /sys 2024-06-06T03:06:59.540 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /proc 2024-06-06T03:06:59.540 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /mnt 2024-06-06T03:06:59.541 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /var/tmp 2024-06-06T03:06:59.541 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /home 2024-06-06T03:06:59.541 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /root 2024-06-06T03:06:59.541 INFO:teuthology.orchestra.run.smithi173.stdout:skipping the directory /tmp 2024-06-06T03:06:59.541 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:00.139 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:07:00.139 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /sys 2024-06-06T03:07:00.139 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /proc 2024-06-06T03:07:00.139 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /mnt 2024-06-06T03:07:00.140 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /var/tmp 2024-06-06T03:07:00.140 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /home 2024-06-06T03:07:00.140 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /root 2024-06-06T03:07:00.140 INFO:teuthology.orchestra.run.smithi153.stdout:skipping the directory /tmp 2024-06-06T03:07:00.140 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:00.840 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T03:07:00.902 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:00.904 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T03:07:01.001 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T03:07:01.001 INFO:teuthology.orchestra.run.smithi173.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:01.393 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 24/26 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-06T03:07:01.455 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:01.457 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T03:07:01.554 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-06T03:07:01.555 INFO:teuthology.orchestra.run.smithi153.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:02.215 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 1/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 5/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 6/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 7/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 8/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 9/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 10/26 2024-06-06T03:07:02.216 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-06T03:07:02.217 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 25/26 2024-06-06T03:07:02.782 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:02.782 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 1/26 2024-06-06T03:07:02.782 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 3/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 4/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 5/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 6/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 7/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x 8/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86 9/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 10/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 11/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 12/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 13/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-06T03:07:02.783 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:17.2.7-992.g6ee68927.el9.x86_64 23/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-06T03:07:02.784 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 25/26 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout:Upgraded: 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.049 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T03:07:03.050 INFO:teuthology.orchestra.run.smithi173.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:03.051 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:03.437 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-test 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout:Upgraded: 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: librados2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-radosgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-selinux-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.499 INFO:teuthology.orchestra.run.smithi153.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: librgw2-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-argparse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.500 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:03.501 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:03.836 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-test 2024-06-06T03:07:04.013 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:33 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:04.133 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test x86_64 2:17.2.7-992.g6ee68927.el9 ceph 48 M 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite x86_64 2:17.2.7-992.g6ee68927.el9 ceph 163 k 2024-06-06T03:07:04.134 INFO:teuthology.orchestra.run.smithi173.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout:Install 4 Packages 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:04.135 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 48 M 2024-06-06T03:07:04.136 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 192 M 2024-06-06T03:07:04.136 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:04.363 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:04.410 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:33 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:04.480 INFO:teuthology.orchestra.run.smithi173.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-06-06T03:07:04.483 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:04.522 INFO:teuthology.orchestra.run.smithi173.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-06T03:07:04.530 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test x86_64 2:17.2.7-992.g6ee68927.el9 ceph 48 M 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite x86_64 2:17.2.7-992.g6ee68927.el9 ceph 163 k 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:04.531 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout:Install 4 Packages 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 48 M 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 192 M 2024-06-06T03:07:04.532 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:04.655 INFO:teuthology.orchestra.run.smithi173.stdout:(3/4): libcephsqlite-17.2.7-992.g6ee68927.el9.x 437 kB/s | 163 kB 00:00 2024-06-06T03:07:04.657 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] socat-1.7.4.1-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/socat-1.7.4.1-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:04.774 INFO:teuthology.orchestra.run.smithi153.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 305 kB 00:00 2024-06-06T03:07:04.777 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] xmlstarlet-1.6.1-20.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/xmlstarlet-1.6.1-20.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:04.816 INFO:teuthology.orchestra.run.smithi153.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-06T03:07:04.841 INFO:teuthology.orchestra.run.smithi153.stdout:(3/4): libcephsqlite-17.2.7-992.g6ee68927.el9.x 619 kB/s | 163 kB 00:00 2024-06-06T03:07:07.189 INFO:teuthology.orchestra.run.smithi173.stdout:(4/4): ceph-test-17.2.7-992.g6ee68927.el9.x86_6 16 MB/s | 48 MB 00:02 2024-06-06T03:07:07.192 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:07.192 INFO:teuthology.orchestra.run.smithi173.stdout:Total 16 MB/s | 48 MB 00:03 2024-06-06T03:07:07.220 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:07.239 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:07.239 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:07.606 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:07.607 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:08.139 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:08.214 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T03:07:08.254 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T03:07:08.282 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T03:07:08.458 INFO:teuthology.orchestra.run.smithi153.stdout:(4/4): ceph-test-17.2.7-992.g6ee68927.el9.x86_6 12 MB/s | 48 MB 00:03 2024-06-06T03:07:08.460 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:08.461 INFO:teuthology.orchestra.run.smithi153.stdout:Total 12 MB/s | 48 MB 00:03 2024-06-06T03:07:08.488 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:08.507 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:08.507 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:08.874 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:08.874 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:09.401 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:09.476 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-06T03:07:09.515 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-06T03:07:09.543 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T03:07:12.690 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T03:07:12.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T03:07:14.162 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 3/4 2024-06-06T03:07:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T03:07:15.085 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T03:07:15.085 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 1/4 2024-06-06T03:07:15.085 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2/4 2024-06-06T03:07:15.085 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T03:07:15.368 INFO:teuthology.orchestra.run.smithi173.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T03:07:15.369 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:15.369 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:15.655 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph 2024-06-06T03:07:16.257 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:45 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:16.380 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph x86_64 2:17.2.7-992.g6ee68927.el9 ceph 6.5 k 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.1 M 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr x86_64 2:17.2.7-992.g6ee68927.el9 ceph 1.4 M 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 240 k 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.9 M 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 15 M 2024-06-06T03:07:16.383 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T03:07:16.384 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T03:07:16.385 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout:Install 33 Packages 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:16.386 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 27 M 2024-06-06T03:07:16.387 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 96 M 2024-06-06T03:07:16.387 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 4/4 2024-06-06T03:07:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 1/4 2024-06-06T03:07:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2/4 2024-06-06T03:07:16.392 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-06T03:07:16.666 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-06T03:07:16.666 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-test-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout: libcephsqlite-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:16.667 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:16.982 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph 2024-06-06T03:07:17.207 INFO:teuthology.orchestra.run.smithi173.stdout:(1/33): ceph-17.2.7-992.g6ee68927.el9.x86_64.rp 37 kB/s | 6.5 kB 00:00 2024-06-06T03:07:17.423 INFO:teuthology.orchestra.run.smithi173.stdout:(2/33): ceph-mds-17.2.7-992.g6ee68927.el9.x86_6 5.3 MB/s | 2.1 MB 00:00 2024-06-06T03:07:17.561 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:00:46 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:17.684 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:17.686 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:17.686 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph x86_64 2:17.2.7-992.g6ee68927.el9 ceph 6.5 k 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.1 M 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr x86_64 2:17.2.7-992.g6ee68927.el9 ceph 1.4 M 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 240 k 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon x86_64 2:17.2.7-992.g6ee68927.el9 ceph 3.9 M 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 15 M 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-06T03:07:17.687 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-06T03:07:17.688 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:17.689 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Install 33 Packages 2024-06-06T03:07:17.690 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 27 M 2024-06-06T03:07:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 96 M 2024-06-06T03:07:17.690 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:18.090 INFO:teuthology.orchestra.run.smithi173.stdout:(3/33): ceph-mgr-17.2.7-992.g6ee68927.el9.x86_6 1.3 MB/s | 1.4 MB 00:01 2024-06-06T03:07:18.233 INFO:teuthology.orchestra.run.smithi173.stdout:(4/33): ceph-osd-17.2.7-992.g6ee68927.el9.x86_6 19 MB/s | 15 MB 00:00 2024-06-06T03:07:18.257 INFO:teuthology.orchestra.run.smithi173.stdout:(5/33): ceph-mgr-modules-core-17.2.7-992.g6ee68 1.4 MB/s | 240 kB 00:00 2024-06-06T03:07:18.358 INFO:teuthology.orchestra.run.smithi173.stdout:(6/33): ceph-mon-17.2.7-992.g6ee68927.el9.x86_6 3.4 MB/s | 3.9 MB 00:01 2024-06-06T03:07:18.384 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.402 INFO:teuthology.orchestra.run.smithi153.stdout:(1/33): ceph-17.2.7-992.g6ee68927.el9.x86_64.rp 27 kB/s | 6.5 kB 00:00 2024-06-06T03:07:18.416 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.424 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.500 INFO:teuthology.orchestra.run.smithi173.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 949 kB/s | 253 kB 00:00 2024-06-06T03:07:18.525 INFO:teuthology.orchestra.run.smithi173.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 755 kB/s | 106 kB 00:00 2024-06-06T03:07:18.544 INFO:teuthology.orchestra.run.smithi153.stdout:(2/33): ceph-mds-17.2.7-992.g6ee68927.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-06-06T03:07:18.550 INFO:teuthology.orchestra.run.smithi173.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-06-06T03:07:18.576 INFO:teuthology.orchestra.run.smithi173.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-06-06T03:07:18.577 INFO:teuthology.orchestra.run.smithi153.stdout:(3/33): ceph-mgr-17.2.7-992.g6ee68927.el9.x86_6 3.4 MB/s | 1.4 MB 00:00 2024-06-06T03:07:18.601 INFO:teuthology.orchestra.run.smithi173.stdout:(11/33): python3-urllib3-1.26.5-5.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-06-06T03:07:18.601 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.634 INFO:teuthology.orchestra.run.smithi173.stdout:(12/33): python3-cryptography-36.0.1-4.el9.x86_ 3.5 MB/s | 1.2 MB 00:00 2024-06-06T03:07:18.635 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.637 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.678 INFO:teuthology.orchestra.run.smithi153.stdout:(4/33): ceph-mgr-modules-core-17.2.7-992.g6ee68 2.3 MB/s | 240 kB 00:00 2024-06-06T03:07:18.719 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-cffi-1.14.5-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cffi-1.14.5-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:18.743 INFO:teuthology.orchestra.run.smithi173.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 245 kB/s | 35 kB 00:00 2024-06-06T03:07:18.768 INFO:teuthology.orchestra.run.smithi173.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 381 kB/s | 51 kB 00:00 2024-06-06T03:07:18.794 INFO:teuthology.orchestra.run.smithi173.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-06-06T03:07:18.819 INFO:teuthology.orchestra.run.smithi173.stdout:(16/33): python3-mako-1.1.4-6.el9.noarch.rpm 708 kB/s | 172 kB 00:00 2024-06-06T03:07:18.844 INFO:teuthology.orchestra.run.smithi173.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 575 kB/s | 43 kB 00:00 2024-06-06T03:07:18.869 INFO:teuthology.orchestra.run.smithi173.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 427 kB/s | 11 kB 00:00 2024-06-06T03:07:18.903 INFO:teuthology.orchestra.run.smithi173.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-06-06T03:07:18.930 INFO:teuthology.orchestra.run.smithi173.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 290 kB/s | 18 kB 00:00 2024-06-06T03:07:18.958 INFO:teuthology.orchestra.run.smithi173.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 439 kB/s | 23 kB 00:00 2024-06-06T03:07:18.981 INFO:teuthology.orchestra.run.smithi173.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 385 kB/s | 19 kB 00:00 2024-06-06T03:07:19.007 INFO:teuthology.orchestra.run.smithi173.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-06-06T03:07:19.032 INFO:teuthology.orchestra.run.smithi173.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-06-06T03:07:19.057 INFO:teuthology.orchestra.run.smithi173.stdout:(25/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-06-06T03:07:19.083 INFO:teuthology.orchestra.run.smithi173.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-06-06T03:07:19.109 INFO:teuthology.orchestra.run.smithi173.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-06-06T03:07:19.133 INFO:teuthology.orchestra.run.smithi173.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-06-06T03:07:19.158 INFO:teuthology.orchestra.run.smithi173.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-06-06T03:07:19.192 INFO:teuthology.orchestra.run.smithi173.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-06-06T03:07:19.225 INFO:teuthology.orchestra.run.smithi173.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2024-06-06T03:07:19.251 INFO:teuthology.orchestra.run.smithi173.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-06-06T03:07:19.276 INFO:teuthology.orchestra.run.smithi173.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-06-06T03:07:19.280 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:19.280 INFO:teuthology.orchestra.run.smithi173.stdout:Total 9.4 MB/s | 27 MB 00:02 2024-06-06T03:07:19.336 INFO:teuthology.orchestra.run.smithi153.stdout:(5/33): ceph-osd-17.2.7-992.g6ee68927.el9.x86_6 19 MB/s | 15 MB 00:00 2024-06-06T03:07:19.339 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-cryptography-36.0.1-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-cryptography-36.0.1-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:19.394 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:19.403 INFO:teuthology.orchestra.run.smithi153.stdout:(6/33): ceph-mon-17.2.7-992.g6ee68927.el9.x86_6 3.9 MB/s | 3.9 MB 00:01 2024-06-06T03:07:19.406 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-ply-3.11-14.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/python3-ply-3.11-14.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:19.431 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:19.431 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:19.807 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:19.808 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:20.952 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:21.004 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T03:07:21.036 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T03:07:21.069 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T03:07:21.162 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T03:07:21.249 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T03:07:21.298 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T03:07:21.333 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T03:07:21.363 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T03:07:21.397 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T03:07:21.481 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T03:07:21.522 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T03:07:21.564 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T03:07:21.602 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T03:07:21.637 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T03:07:21.711 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T03:07:21.858 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T03:07:21.938 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T03:07:21.991 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T03:07:22.030 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T03:07:22.087 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T03:07:22.147 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T03:07:22.348 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T03:07:22.400 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T03:07:22.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T03:07:22.627 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T03:07:22.667 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T03:07:22.784 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T03:07:22.946 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 28/33 2024-06-06T03:07:22.977 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T03:07:23.003 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:23.336 INFO:teuthology.orchestra.run.smithi153.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 27 kB/s | 106 kB 00:03 2024-06-06T03:07:23.445 INFO:teuthology.orchestra.run.smithi153.stdout:(8/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 53 kB/s | 253 kB 00:04 2024-06-06T03:07:23.487 INFO:teuthology.orchestra.run.smithi153.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 901 kB/s | 135 kB 00:00 2024-06-06T03:07:23.520 INFO:teuthology.orchestra.run.smithi153.stdout:(10/33): python3-cryptography-36.0.1-4.el9.x86_ 300 kB/s | 1.2 MB 00:04 2024-06-06T03:07:23.523 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-mako-1.1.4-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-mako-1.1.4-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:23.554 INFO:teuthology.orchestra.run.smithi153.stdout:(11/33): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-06-06T03:07:23.555 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-markupsafe-1.1.1-12.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-markupsafe-1.1.1-12.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:23.612 INFO:teuthology.orchestra.run.smithi153.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 598 kB/s | 35 kB 00:00 2024-06-06T03:07:23.613 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-pytz-2021.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pytz-2021.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:23.638 INFO:teuthology.orchestra.run.smithi153.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.4 MB/s | 215 kB 00:00 2024-06-06T03:07:23.663 INFO:teuthology.orchestra.run.smithi153.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-06-06T03:07:23.697 INFO:teuthology.orchestra.run.smithi153.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 608 kB/s | 51 kB 00:00 2024-06-06T03:07:23.722 INFO:teuthology.orchestra.run.smithi153.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 742 kB/s | 43 kB 00:00 2024-06-06T03:07:23.772 INFO:teuthology.orchestra.run.smithi153.stdout:(17/33): python3-toml-0.10.2-6.el9.noarch.rpm 311 kB/s | 42 kB 00:00 2024-06-06T03:07:23.839 INFO:teuthology.orchestra.run.smithi153.stdout:(18/33): python3-cherrypy-18.6.1-2.el9.noarch.r 3.9 MB/s | 358 kB 00:00 2024-06-06T03:07:23.864 INFO:teuthology.orchestra.run.smithi153.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-06-06T03:07:23.890 INFO:teuthology.orchestra.run.smithi153.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 93 kB/s | 11 kB 00:00 2024-06-06T03:07:23.915 INFO:teuthology.orchestra.run.smithi153.stdout:(21/33): python3-jaraco-classes-3.2.1-5.el9.noa 241 kB/s | 18 kB 00:00 2024-06-06T03:07:23.940 INFO:teuthology.orchestra.run.smithi153.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-06-06T03:07:23.965 INFO:teuthology.orchestra.run.smithi153.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-06-06T03:07:24.007 INFO:teuthology.orchestra.run.smithi153.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-06-06T03:07:24.033 INFO:teuthology.orchestra.run.smithi153.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 692 kB/s | 46 kB 00:00 2024-06-06T03:07:24.058 INFO:teuthology.orchestra.run.smithi153.stdout:(26/33): python3-more-itertools-8.12.0-2.el9.no 856 kB/s | 79 kB 00:00 2024-06-06T03:07:24.083 INFO:teuthology.orchestra.run.smithi153.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-06-06T03:07:24.109 INFO:teuthology.orchestra.run.smithi153.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-06-06T03:07:24.142 INFO:teuthology.orchestra.run.smithi153.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-06-06T03:07:24.168 INFO:teuthology.orchestra.run.smithi153.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-06-06T03:07:24.201 INFO:teuthology.orchestra.run.smithi153.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 601 kB/s | 20 kB 00:00 2024-06-06T03:07:24.235 INFO:teuthology.orchestra.run.smithi153.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-06-06T03:07:24.293 INFO:teuthology.orchestra.run.smithi153.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-06-06T03:07:24.298 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:24.298 INFO:teuthology.orchestra.run.smithi153.stdout:Total 4.1 MB/s | 27 MB 00:06 2024-06-06T03:07:24.387 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T03:07:24.418 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:24.422 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T03:07:24.422 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:24.423 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T03:07:24.423 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T03:07:24.423 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T03:07:24.425 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:24.455 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:24.455 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:24.796 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T03:07:24.820 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T03:07:24.820 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:24.821 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T03:07:24.821 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T03:07:24.821 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T03:07:24.821 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:24.821 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:24.822 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:25.022 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T03:07:25.046 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T03:07:25.046 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:25.046 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T03:07:25.046 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T03:07:25.046 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T03:07:25.047 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:25.112 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T03:07:25.980 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:26.046 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-06T03:07:26.070 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-06T03:07:26.103 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-06T03:07:26.194 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-06T03:07:26.272 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-06T03:07:26.313 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-06T03:07:26.348 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-06T03:07:26.378 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-06T03:07:26.413 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-06T03:07:26.446 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-06T03:07:26.487 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-06T03:07:26.522 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-06T03:07:26.557 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-06T03:07:26.595 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-06T03:07:26.667 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-06T03:07:26.760 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T03:07:26.760 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 1/33 2024-06-06T03:07:26.760 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2/33 2024-06-06T03:07:26.760 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 3/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 4/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 5/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 6/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T03:07:26.761 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T03:07:26.762 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T03:07:26.763 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T03:07:26.763 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T03:07:26.763 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T03:07:26.764 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T03:07:26.764 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T03:07:26.812 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-06T03:07:26.903 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-06T03:07:26.957 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-06T03:07:26.995 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-06T03:07:27.052 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-06T03:07:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-06T03:07:27.306 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-06T03:07:27.349 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-06T03:07:27.428 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-06T03:07:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-06T03:07:27.549 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-06T03:07:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T03:07:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:27.612 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:27.612 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T03:07:27.613 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T03:07:27.614 INFO:teuthology.orchestra.run.smithi173.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T03:07:27.615 INFO:teuthology.orchestra.run.smithi173.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T03:07:27.616 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:27.616 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-06T03:07:27.811 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 28/33 2024-06-06T03:07:27.842 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T03:07:27.867 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 29/33 2024-06-06T03:07:27.868 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:27.868 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-06T03:07:27.868 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T03:07:27.868 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-06T03:07:27.868 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:28.000 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-base 2024-06-06T03:07:28.580 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:57 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:28.661 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:07:28.703 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:28.704 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:07:28.705 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:28.779 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install cephadm 2024-06-06T03:07:29.193 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 30/33 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-06T03:07:29.221 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:29.350 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:00:58 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:29.473 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 82 k 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:29.474 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 82 k 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 372 k 2024-06-06T03:07:29.475 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:29.620 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T03:07:29.645 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 31/33 2024-06-06T03:07:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-06T03:07:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T03:07:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-06T03:07:29.646 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:29.800 INFO:teuthology.orchestra.run.smithi173.stdout:cephadm-17.2.7-992.g6ee68927.el9.noarch.rpm 252 kB/s | 82 kB 00:00 2024-06-06T03:07:29.800 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:29.801 INFO:teuthology.orchestra.run.smithi173.stdout:Total 251 kB/s | 82 kB 00:00 2024-06-06T03:07:29.801 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:29.806 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:29.806 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:29.808 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:29.809 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:29.854 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 32/33 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-06T03:07:29.880 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:29.885 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:29.960 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T03:07:30.308 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:30.428 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:30.762 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:31.003 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:31.191 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T03:07:31.541 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 33/33 2024-06-06T03:07:31.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-2:17.2.7-992.g6ee68927.el9.x86_64 1/33 2024-06-06T03:07:31.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2/33 2024-06-06T03:07:31.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 3/33 2024-06-06T03:07:31.541 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 4/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 5/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.n 6/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-06T03:07:31.542 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-06T03:07:31.543 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-06T03:07:31.544 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-06T03:07:31.786 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:00 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:31.910 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-992.g6ee68927.el9 ceph 138 k 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:31.911 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 138 k 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 379 k 2024-06-06T03:07:31.912 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:ceph-immutable-object-cache-17.2.7-992.g6ee6892 388 kB/s | 138 kB 00:00 2024-06-06T03:07:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:32.267 INFO:teuthology.orchestra.run.smithi173.stdout:Total 386 kB/s | 138 kB 00:00 2024-06-06T03:07:32.268 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:32.275 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:32.275 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:32.332 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:32.332 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mds-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mon-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-osd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:32.473 INFO:teuthology.orchestra.run.smithi153.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-06T03:07:32.474 INFO:teuthology.orchestra.run.smithi153.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ply-3.11-14.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-06T03:07:32.475 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-06T03:07:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-06T03:07:32.476 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:32.476 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:32.481 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:32.576 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:32.602 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:32.602 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:32.602 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T03:07:32.602 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:32.835 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-base 2024-06-06T03:07:33.417 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:02 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:33.434 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:33.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:33.435 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:33.435 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:33.435 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:33.435 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:33.498 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-base-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:07:33.541 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:33.542 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:07:33.542 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:33.611 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install cephadm 2024-06-06T03:07:33.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr 2024-06-06T03:07:34.176 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:03 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:34.258 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:03 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:34.300 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 82 k 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:34.301 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:34.302 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:07:34.302 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:34.302 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-06-06T03:07:34.302 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 372 k 2024-06-06T03:07:34.302 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:34.343 INFO:teuthology.orchestra.run.smithi173.stdout:Package ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:07:34.386 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:34.387 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:07:34.387 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:34.455 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T03:07:34.613 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm-17.2.7-992.g6ee68927.el9.noarch.rpm 264 kB/s | 82 kB 00:00 2024-06-06T03:07:34.613 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:34.613 INFO:teuthology.orchestra.run.smithi153.stdout:Total 263 kB/s | 82 kB 00:00 2024-06-06T03:07:34.614 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:34.618 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:34.619 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:34.620 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:34.621 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:34.693 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:35.042 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:04 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:35.137 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:35.164 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 3.4 M 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 24 k 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 15 k 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T03:07:35.165 INFO:teuthology.orchestra.run.smithi173.stdout:Installing weak dependencies: 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout:Install 7 Packages 2024-06-06T03:07:35.166 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:35.167 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 3.7 M 2024-06-06T03:07:35.167 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 80 M 2024-06-06T03:07:35.167 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:35.241 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:35.464 INFO:teuthology.orchestra.run.smithi173.stdout:(1/7): ceph-grafana-dashboards-17.2.7-992.g6ee6 97 kB/s | 24 kB 00:00 2024-06-06T03:07:35.531 INFO:teuthology.orchestra.run.smithi173.stdout:(2/7): ceph-prometheus-alerts-17.2.7-992.g6ee68 46 kB/s | 15 kB 00:00 2024-06-06T03:07:35.556 INFO:teuthology.orchestra.run.smithi173.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 98 kB/s | 9.0 kB 00:00 2024-06-06T03:07:35.574 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:35.615 INFO:teuthology.orchestra.run.smithi173.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 879 kB/s | 31 kB 00:00 2024-06-06T03:07:35.640 INFO:teuthology.orchestra.run.smithi173.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 375 kB/s | 41 kB 00:00 2024-06-06T03:07:35.699 INFO:teuthology.orchestra.run.smithi173.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-06-06T03:07:35.838 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/1 2024-06-06T03:07:35.838 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:35.839 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:35.839 INFO:teuthology.orchestra.run.smithi153.stdout: cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:35.839 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:35.839 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:35.857 INFO:teuthology.orchestra.run.smithi173.stdout:(7/7): ceph-mgr-dashboard-17.2.7-992.g6ee68927. 5.2 MB/s | 3.4 MB 00:00 2024-06-06T03:07:35.861 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:35.861 INFO:teuthology.orchestra.run.smithi173.stdout:Total 5.3 MB/s | 3.7 MB 00:00 2024-06-06T03:07:35.861 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:35.869 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:35.869 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:35.931 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:35.931 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:36.060 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-immutable-object-cache 2024-06-06T03:07:36.234 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:36.292 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-06T03:07:36.326 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-06T03:07:36.372 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-06T03:07:36.413 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-06T03:07:36.442 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 5/7 2024-06-06T03:07:36.642 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:05 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:36.767 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-992.g6ee68927.el9 ceph 138 k 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:36.768 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 138 k 2024-06-06T03:07:36.769 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 379 k 2024-06-06T03:07:36.769 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:37.119 INFO:teuthology.orchestra.run.smithi153.stdout:ceph-immutable-object-cache-17.2.7-992.g6ee6892 393 kB/s | 138 kB 00:00 2024-06-06T03:07:37.120 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:37.120 INFO:teuthology.orchestra.run.smithi153.stdout:Total 391 kB/s | 138 kB 00:00 2024-06-06T03:07:37.120 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:37.127 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:37.127 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:37.184 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:37.184 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:37.331 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:37.395 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:37.420 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:37.420 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:07:37.420 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-06T03:07:37.420 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:37.435 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 6/7 2024-06-06T03:07:37.488 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T03:07:38.304 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.e 1/1 2024-06-06T03:07:38.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:38.305 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:38.305 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-immutable-object-cache-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:07:38.305 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:38.305 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:38.496 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr 2024-06-06T03:07:38.625 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T03:07:38.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 1/7 2024-06-06T03:07:38.625 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2/7 2024-06-06T03:07:38.626 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 3/7 2024-06-06T03:07:38.626 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-06T03:07:38.626 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-06T03:07:38.626 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-06T03:07:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-06T03:07:38.957 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:38.957 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:38.958 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:39.073 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:08 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:39.146 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T03:07:39.159 INFO:teuthology.orchestra.run.smithi153.stdout:Package ceph-mgr-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:07:39.202 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:39.203 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:07:39.203 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:39.265 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-dashboard 2024-06-06T03:07:39.711 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:08 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:39.826 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:08 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:39.833 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:39.834 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-06-06T03:07:39.834 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:39.834 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-06-06T03:07:39.834 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:39.834 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 7.4 M 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T03:07:39.835 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:====================================================================================== 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:Install 12 Packages 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 38 M 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 209 M 2024-06-06T03:07:39.836 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:39.955 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 3.4 M 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 24 k 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 15 k 2024-06-06T03:07:39.956 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout:Installing weak dependencies: 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout:Install 7 Packages 2024-06-06T03:07:39.957 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:39.958 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 3.7 M 2024-06-06T03:07:39.958 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 80 M 2024-06-06T03:07:39.958 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:40.206 INFO:teuthology.orchestra.run.smithi153.stdout:(1/7): ceph-prometheus-alerts-17.2.7-992.g6ee68 73 kB/s | 15 kB 00:00 2024-06-06T03:07:40.231 INFO:teuthology.orchestra.run.smithi153.stdout:(2/7): ceph-grafana-dashboards-17.2.7-992.g6ee6 109 kB/s | 24 kB 00:00 2024-06-06T03:07:40.313 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:40.315 INFO:teuthology.orchestra.run.smithi153.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-06-06T03:07:40.315 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:40.356 INFO:teuthology.orchestra.run.smithi153.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 783 kB/s | 31 kB 00:00 2024-06-06T03:07:40.382 INFO:teuthology.orchestra.run.smithi153.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 272 kB/s | 41 kB 00:00 2024-06-06T03:07:40.467 INFO:teuthology.orchestra.run.smithi173.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 837 kB/s | 195 kB 00:00 2024-06-06T03:07:40.470 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:40.473 INFO:teuthology.orchestra.run.smithi153.stdout:(6/7): ceph-mgr-dashboard-17.2.7-992.g6ee68927. 7.2 MB/s | 3.4 MB 00:00 2024-06-06T03:07:40.507 INFO:teuthology.orchestra.run.smithi153.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.2 MB/s | 188 kB 00:00 2024-06-06T03:07:40.511 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:40.511 INFO:teuthology.orchestra.run.smithi153.stdout:Total 6.6 MB/s | 3.7 MB 00:00 2024-06-06T03:07:40.512 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:40.521 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:40.521 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:40.542 INFO:teuthology.orchestra.run.smithi173.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 437 kB/s | 33 kB 00:00 2024-06-06T03:07:40.567 INFO:teuthology.orchestra.run.smithi173.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-06-06T03:07:40.568 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:40.570 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:40.586 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:40.588 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:40.626 INFO:teuthology.orchestra.run.smithi173.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 311 kB/s | 18 kB 00:00 2024-06-06T03:07:40.693 INFO:teuthology.orchestra.run.smithi173.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 529 kB/s | 35 kB 00:00 2024-06-06T03:07:40.752 INFO:teuthology.orchestra.run.smithi173.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-06-06T03:07:40.795 INFO:teuthology.orchestra.run.smithi173.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-06-06T03:07:40.886 INFO:teuthology.orchestra.run.smithi173.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-06-06T03:07:40.891 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:40.961 INFO:teuthology.orchestra.run.smithi173.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 31 MB/s | 5.1 MB 00:00 2024-06-06T03:07:40.967 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-06T03:07:40.994 INFO:teuthology.orchestra.run.smithi173.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.3 MB/s | 250 kB 00:00 2024-06-06T03:07:41.001 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-06T03:07:41.056 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-06T03:07:41.087 INFO:teuthology.orchestra.run.smithi173.stdout:(11/12): ceph-mgr-diskprediction-local-17.2.7-9 8.7 MB/s | 7.4 MB 00:00 2024-06-06T03:07:41.088 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-06T03:07:41.117 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 5/7 2024-06-06T03:07:41.329 INFO:teuthology.orchestra.run.smithi173.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-06-06T03:07:41.331 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:41.331 INFO:teuthology.orchestra.run.smithi173.stdout:Total 25 MB/s | 38 MB 00:01 2024-06-06T03:07:41.678 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:41.704 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:41.704 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:42.119 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 6/7 2024-06-06T03:07:42.162 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T03:07:42.213 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:42.214 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:42.979 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:43.048 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T03:07:43.102 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T03:07:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 7/7 2024-06-06T03:07:43.118 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.n 1/7 2024-06-06T03:07:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2/7 2024-06-06T03:07:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.no 3/7 2024-06-06T03:07:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-06T03:07:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-06T03:07:43.119 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-06T03:07:43.201 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T03:07:43.245 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T03:07:43.426 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-grafana-dashboards-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-prometheus-alerts-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:43.429 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:43.461 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T03:07:43.547 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T03:07:43.642 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-06T03:07:44.208 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:13 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:44.215 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T03:07:44.331 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 7.4 M 2024-06-06T03:07:44.332 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-06T03:07:44.333 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:44.334 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:44.334 INFO:teuthology.orchestra.run.smithi153.stdout:====================================================================================== 2024-06-06T03:07:44.334 INFO:teuthology.orchestra.run.smithi153.stdout:Install 12 Packages 2024-06-06T03:07:44.334 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:44.334 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 38 M 2024-06-06T03:07:44.335 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 209 M 2024-06-06T03:07:44.335 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:44.349 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T03:07:44.677 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:44.678 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:44.790 INFO:teuthology.orchestra.run.smithi153.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-06-06T03:07:44.793 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] flexiblas-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:44.832 INFO:teuthology.orchestra.run.smithi153.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 785 kB/s | 33 kB 00:00 2024-06-06T03:07:44.857 INFO:teuthology.orchestra.run.smithi153.stdout:(3/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.0 MB/s | 803 kB 00:00 2024-06-06T03:07:44.858 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] flexiblas-netlib-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-netlib-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:44.861 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/flexiblas-openblas-openmp-3.0.4-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:07:44.899 INFO:teuthology.orchestra.run.smithi153.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-06-06T03:07:44.942 INFO:teuthology.orchestra.run.smithi153.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 35 kB 00:00 2024-06-06T03:07:44.992 INFO:teuthology.orchestra.run.smithi153.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 19 MB/s | 3.0 MB 00:00 2024-06-06T03:07:45.034 INFO:teuthology.orchestra.run.smithi153.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 5.7 MB/s | 244 kB 00:00 2024-06-06T03:07:45.117 INFO:teuthology.orchestra.run.smithi153.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-06-06T03:07:45.151 INFO:teuthology.orchestra.run.smithi153.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 7.4 MB/s | 250 kB 00:00 2024-06-06T03:07:45.235 INFO:teuthology.orchestra.run.smithi153.stdout:(10/12): ceph-mgr-diskprediction-local-17.2.7-9 12 MB/s | 7.4 MB 00:00 2024-06-06T03:07:45.293 INFO:teuthology.orchestra.run.smithi153.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-06-06T03:07:45.560 INFO:teuthology.orchestra.run.smithi153.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-06-06T03:07:45.563 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:45.563 INFO:teuthology.orchestra.run.smithi153.stdout:Total 31 MB/s | 38 MB 00:01 2024-06-06T03:07:45.575 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T03:07:45.915 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:45.940 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:45.940 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:46.434 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:46.435 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:46.614 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T03:07:46.665 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T03:07:47.191 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:47.261 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-06T03:07:47.300 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-06T03:07:47.399 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-06T03:07:47.451 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-06T03:07:47.638 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-06T03:07:47.675 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-06T03:07:47.752 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-06T03:07:48.411 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-06T03:07:48.522 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-06T03:07:49.324 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T03:07:49.324 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 1/12 2024-06-06T03:07:49.324 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T03:07:49.325 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T03:07:49.717 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T03:07:49.718 INFO:teuthology.orchestra.run.smithi173.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:49.719 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:49.771 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T03:07:50.161 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-rook 2024-06-06T03:07:50.750 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:19 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:50.789 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-06T03:07:50.817 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T03:07:50.875 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:50.876 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:50.876 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:50.876 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:50.876 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 47 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T03:07:50.877 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout:Install 13 Packages 2024-06-06T03:07:50.878 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:50.879 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.2 M 2024-06-06T03:07:50.879 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 26 M 2024-06-06T03:07:50.879 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:51.125 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:51.127 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:51.212 INFO:teuthology.orchestra.run.smithi173.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 152 kB/s | 26 kB 00:00 2024-06-06T03:07:51.237 INFO:teuthology.orchestra.run.smithi173.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 19 kB 00:00 2024-06-06T03:07:51.238 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:51.241 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:51.321 INFO:teuthology.orchestra.run.smithi173.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-06-06T03:07:51.346 INFO:teuthology.orchestra.run.smithi173.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-06-06T03:07:51.372 INFO:teuthology.orchestra.run.smithi173.stdout:(5/13): ceph-mgr-rook-17.2.7-992.g6ee68927.el9. 145 kB/s | 47 kB 00:00 2024-06-06T03:07:51.397 INFO:teuthology.orchestra.run.smithi173.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-06T03:07:51.422 INFO:teuthology.orchestra.run.smithi173.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 636 kB/s | 32 kB 00:00 2024-06-06T03:07:51.448 INFO:teuthology.orchestra.run.smithi173.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 298 kB/s | 15 kB 00:00 2024-06-06T03:07:51.481 INFO:teuthology.orchestra.run.smithi173.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.7 MB/s | 279 kB 00:00 2024-06-06T03:07:51.506 INFO:teuthology.orchestra.run.smithi173.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 2.4 MB/s | 206 kB 00:00 2024-06-06T03:07:51.582 INFO:teuthology.orchestra.run.smithi173.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-06-06T03:07:51.607 INFO:teuthology.orchestra.run.smithi173.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-06-06T03:07:51.774 INFO:teuthology.orchestra.run.smithi173.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-06-06T03:07:51.777 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:51.777 INFO:teuthology.orchestra.run.smithi173.stdout:Total 2.4 MB/s | 2.2 MB 00:00 2024-06-06T03:07:51.843 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:51.858 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:51.858 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:51.943 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:51.943 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:52.451 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:52.621 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T03:07:52.675 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T03:07:52.730 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T03:07:52.768 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T03:07:52.809 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T03:07:52.902 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T03:07:53.013 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T03:07:53.061 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T03:07:53.317 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 12/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee689 1/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-06T03:07:53.318 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-06T03:07:53.721 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-diskprediction-local-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-06T03:07:53.722 INFO:teuthology.orchestra.run.smithi153.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-06T03:07:53.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-06T03:07:53.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-06T03:07:53.723 INFO:teuthology.orchestra.run.smithi153.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-06T03:07:53.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:53.723 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:53.787 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T03:07:53.833 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T03:07:53.875 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T03:07:53.920 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T03:07:53.944 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T03:07:54.051 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-rook 2024-06-06T03:07:54.638 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:23 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 1/13 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T03:07:54.683 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T03:07:54.684 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T03:07:54.761 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:54.762 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 47 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-06T03:07:54.763 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout:Install 13 Packages 2024-06-06T03:07:54.764 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:54.765 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.2 M 2024-06-06T03:07:54.765 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 26 M 2024-06-06T03:07:54.765 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:55.007 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T03:07:55.008 INFO:teuthology.orchestra.run.smithi173.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T03:07:55.009 INFO:teuthology.orchestra.run.smithi173.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T03:07:55.009 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:55.009 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:55.263 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-jsonpatch-1.21-16.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpatch-1.21-16.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:55.266 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-jsonpointer-2.0-4.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jsonpointer-2.0-4.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:55.267 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T03:07:55.326 INFO:teuthology.orchestra.run.smithi153.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 26 kB 00:00 2024-06-06T03:07:55.351 INFO:teuthology.orchestra.run.smithi153.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 19 kB 00:00 2024-06-06T03:07:55.351 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-oauthlib-3.1.1-5.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-oauthlib-3.1.1-5.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:55.352 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-pyasn1-0.4.8-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-pyasn1-0.4.8-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:55.393 INFO:teuthology.orchestra.run.smithi153.stdout:(3/13): ceph-mgr-rook-17.2.7-992.g6ee68927.el9. 226 kB/s | 47 kB 00:00 2024-06-06T03:07:55.443 INFO:teuthology.orchestra.run.smithi153.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 222 kB 00:00 2024-06-06T03:07:55.468 INFO:teuthology.orchestra.run.smithi153.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-06-06T03:07:55.494 INFO:teuthology.orchestra.run.smithi153.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-06T03:07:55.519 INFO:teuthology.orchestra.run.smithi153.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-06-06T03:07:55.544 INFO:teuthology.orchestra.run.smithi153.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 297 kB/s | 15 kB 00:00 2024-06-06T03:07:55.569 INFO:teuthology.orchestra.run.smithi153.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.5 MB/s | 279 kB 00:00 2024-06-06T03:07:55.620 INFO:teuthology.orchestra.run.smithi153.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 2.0 MB/s | 206 kB 00:00 2024-06-06T03:07:55.662 INFO:teuthology.orchestra.run.smithi153.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 8.7 MB/s | 1.0 MB 00:00 2024-06-06T03:07:55.687 INFO:teuthology.orchestra.run.smithi153.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-06-06T03:07:55.712 INFO:teuthology.orchestra.run.smithi153.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-06-06T03:07:55.715 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:55.716 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.3 MB/s | 2.2 MB 00:00 2024-06-06T03:07:55.779 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:07:55.793 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:07:55.794 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:07:55.848 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:24 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:55.878 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:07:55.887 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:07:55.971 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 123 k 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:07:55.972 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout:Install 6 Packages 2024-06-06T03:07:55.973 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:55.974 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 7.0 M 2024-06-06T03:07:55.974 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 33 M 2024-06-06T03:07:55.974 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:07:56.257 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:56.257 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:07:56.380 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:07:56.426 INFO:teuthology.orchestra.run.smithi173.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 997 kB/s | 248 kB 00:00 2024-06-06T03:07:56.477 INFO:teuthology.orchestra.run.smithi173.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-06-06T03:07:56.502 INFO:teuthology.orchestra.run.smithi173.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-06T03:07:56.527 INFO:teuthology.orchestra.run.smithi173.stdout:(4/6): ceph-mgr-cephadm-17.2.7-992.g6ee68927.el 353 kB/s | 123 kB 00:00 2024-06-06T03:07:56.552 INFO:teuthology.orchestra.run.smithi173.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-06T03:07:56.566 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-06T03:07:56.621 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-06T03:07:56.661 INFO:teuthology.orchestra.run.smithi173.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-06-06T03:07:56.663 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:07:56.663 INFO:teuthology.orchestra.run.smithi173.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-06-06T03:07:56.676 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-06T03:07:56.714 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-06T03:07:56.734 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:07:56.746 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:07:56.746 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:07:56.755 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-06T03:07:56.841 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:07:56.842 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:07:56.848 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-06T03:07:56.950 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-06T03:07:57.007 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-06T03:07:57.137 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:07:57.300 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T03:07:57.362 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T03:07:57.725 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-06T03:07:57.771 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-06T03:07:57.804 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-06T03:07:57.872 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T03:07:57.875 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-06T03:07:57.899 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T03:07:57.965 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T03:07:58.020 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T03:07:58.050 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 13/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 1/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-06T03:07:58.683 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-06T03:07:58.684 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-06T03:07:58.684 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-06T03:07:58.684 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-06T03:07:58.684 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-06T03:07:58.684 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/6 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T03:07:58.906 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T03:07:58.995 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-06T03:07:58.995 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-rook-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-06T03:07:58.996 INFO:teuthology.orchestra.run.smithi153.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-06T03:07:58.997 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-06T03:07:58.997 INFO:teuthology.orchestra.run.smithi153.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-06T03:07:58.997 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:58.997 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T03:07:59.174 INFO:teuthology.orchestra.run.smithi173.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T03:07:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T03:07:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T03:07:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T03:07:59.175 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:07:59.175 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:07:59.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-mgr-cephadm 2024-06-06T03:07:59.416 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install ceph-fuse 2024-06-06T03:07:59.846 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:28 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:07:59.970 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:07:59.971 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:07:59.971 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:59.971 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:07:59.971 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm noarch 2:17.2.7-992.g6ee68927.el9 ceph-noarch 123 k 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout:Install 6 Packages 2024-06-06T03:07:59.972 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:07:59.973 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 7.0 M 2024-06-06T03:07:59.973 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 33 M 2024-06-06T03:07:59.973 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:07:59.985 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:28 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:00.108 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:00.108 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:00.108 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repository Size 2024-06-06T03:08:00.108 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 813 k 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout:Installing dependencies: 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout:Install 2 Packages 2024-06-06T03:08:00.109 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 893 k 2024-06-06T03:08:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 2.6 M 2024-06-06T03:08:00.110 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:00.299 INFO:teuthology.orchestra.run.smithi173.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:08:00.309 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-babel-2.9.1-2.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-babel-2.9.1-2.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:08:00.311 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] python3-jinja2-2.11.3-6.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/python3-jinja2-2.11.3-6.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-06T03:08:00.421 INFO:teuthology.orchestra.run.smithi173.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 434 kB/s | 80 kB 00:00 2024-06-06T03:08:00.473 INFO:teuthology.orchestra.run.smithi153.stdout:(1/6): ceph-mgr-cephadm-17.2.7-992.g6ee68927.el 498 kB/s | 123 kB 00:00 2024-06-06T03:08:00.615 INFO:teuthology.orchestra.run.smithi153.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.8 MB/s | 548 kB 00:00 2024-06-06T03:08:00.641 INFO:teuthology.orchestra.run.smithi153.stdout:(3/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 598 kB/s | 248 kB 00:00 2024-06-06T03:08:00.665 INFO:teuthology.orchestra.run.smithi153.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-06-06T03:08:00.690 INFO:teuthology.orchestra.run.smithi153.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-06T03:08:00.704 INFO:teuthology.orchestra.run.smithi173.stdout:(2/2): ceph-fuse-17.2.7-992.g6ee68927.el9.x86_6 1.7 MB/s | 813 kB 00:00 2024-06-06T03:08:00.705 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:00.706 INFO:teuthology.orchestra.run.smithi173.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-06-06T03:08:00.718 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:00.740 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:00.740 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:00.817 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:00.817 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:00.996 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:01.116 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T03:08:01.194 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T03:08:01.846 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T03:08:01.847 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/2 2024-06-06T03:08:02.132 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T03:08:02.132 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:02.133 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:02.133 INFO:teuthology.orchestra.run.smithi173.stdout: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T03:08:02.133 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:02.133 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:02.404 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados-devel 2024-06-06T03:08:02.990 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:31 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:03.117 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:03.117 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:03.117 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-06-06T03:08:03.117 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 126 k 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:08:03.118 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:03.119 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 126 k 2024-06-06T03:08:03.119 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 448 k 2024-06-06T03:08:03.119 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:03.443 INFO:teuthology.orchestra.run.smithi173.stdout:librados-devel-17.2.7-992.g6ee68927.el9.x86_64. 390 kB/s | 126 kB 00:00 2024-06-06T03:08:03.443 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:03.443 INFO:teuthology.orchestra.run.smithi173.stdout:Total 388 kB/s | 126 kB 00:00 2024-06-06T03:08:03.443 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:03.449 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:03.449 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:03.474 INFO:teuthology.orchestra.run.smithi153.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.8 MB/s | 6.0 MB 00:03 2024-06-06T03:08:03.476 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:03.476 INFO:teuthology.orchestra.run.smithi153.stdout:Total 2.0 MB/s | 7.0 MB 00:03 2024-06-06T03:08:03.511 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:03.511 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:03.549 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:03.561 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:03.561 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:03.656 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:03.658 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:03.658 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:03.764 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:03.961 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:04.141 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:04.146 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-06T03:08:04.208 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:04.394 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:04.645 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs2 2024-06-06T03:08:04.737 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-06T03:08:04.828 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-06T03:08:04.892 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-06T03:08:04.929 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T03:08:05.230 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:34 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:05.314 INFO:teuthology.orchestra.run.smithi173.stdout:Package libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:05.356 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:05.357 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:05.357 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:05.438 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install libcephfs-devel 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 6/6 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 1/6 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-06T03:08:05.717 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-cephadm-2:17.2.7-992.g6ee68927.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-06T03:08:06.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:06.002 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:06.016 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:06.139 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:08:06.140 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:06.141 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 25 k 2024-06-06T03:08:06.141 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 110 k 2024-06-06T03:08:06.141 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:06.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install ceph-fuse 2024-06-06T03:08:06.312 INFO:teuthology.orchestra.run.smithi173.stdout:libcephfs-devel-17.2.7-992.g6ee68927.el9.x86_64 145 kB/s | 25 kB 00:00 2024-06-06T03:08:06.313 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:06.313 INFO:teuthology.orchestra.run.smithi173.stdout:Total 144 kB/s | 25 kB 00:00 2024-06-06T03:08:06.313 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:06.317 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:06.318 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:06.319 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:06.393 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:06.500 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:06.820 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:35 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:06.844 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:06.942 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:06.942 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:06.942 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repository Size 2024-06-06T03:08:06.942 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 813 k 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout:Installing dependencies: 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout:Install 2 Packages 2024-06-06T03:08:06.943 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:06.944 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 893 k 2024-06-06T03:08:06.944 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 2.6 M 2024-06-06T03:08:06.944 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:07.079 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:07.079 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:07.080 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:07.080 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:07.276 INFO:teuthology.orchestra.run.smithi153.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-06T03:08:07.312 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librados2 2024-06-06T03:08:07.383 INFO:teuthology.orchestra.run.smithi153.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 464 kB/s | 80 kB 00:00 2024-06-06T03:08:07.642 INFO:teuthology.orchestra.run.smithi153.stdout:(2/2): ceph-fuse-17.2.7-992.g6ee68927.el9.x86_6 1.8 MB/s | 813 kB 00:00 2024-06-06T03:08:07.643 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:07.643 INFO:teuthology.orchestra.run.smithi153.stdout:Total 1.2 MB/s | 893 kB 00:00 2024-06-06T03:08:07.655 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:07.676 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:07.676 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:07.754 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:07.754 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:07.883 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:36 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:07.937 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:07.964 INFO:teuthology.orchestra.run.smithi173.stdout:Package librados2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:08.005 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:08.007 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:08.007 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:08.078 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install librbd1 2024-06-06T03:08:08.079 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-06T03:08:08.149 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T03:08:08.647 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:37 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:08.727 INFO:teuthology.orchestra.run.smithi173.stdout:Package librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:08.769 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:08.770 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:08.770 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2/2 2024-06-06T03:08:08.839 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/2 2024-06-06T03:08:08.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rados 2024-06-06T03:08:09.124 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-06T03:08:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:09.125 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-06T03:08:09.125 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:09.125 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:09.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados-devel 2024-06-06T03:08:09.409 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:38 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:09.491 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:09.532 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:09.533 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:09.533 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:09.600 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rgw 2024-06-06T03:08:09.966 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:38 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:10.092 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 126 k 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:10.093 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 126 k 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 448 k 2024-06-06T03:08:10.094 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:10.175 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:10.257 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:10.298 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:10.299 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:10.299 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:10.330 INFO:teuthology.orchestra.run.smithi153.stdout:librados-devel-17.2.7-992.g6ee68927.el9.x86_64. 536 kB/s | 126 kB 00:00 2024-06-06T03:08:10.330 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:10.331 INFO:teuthology.orchestra.run.smithi153.stdout:Total 532 kB/s | 126 kB 00:00 2024-06-06T03:08:10.331 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:10.336 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:10.336 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:10.378 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-cephfs 2024-06-06T03:08:10.399 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:10.399 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:10.545 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:10.643 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:10.947 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:39 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:11.029 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:11.071 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:11.071 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:11.072 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:11.092 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:11.143 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install python3-rbd 2024-06-06T03:08:11.357 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:11.357 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:11.357 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:11.357 INFO:teuthology.orchestra.run.smithi153.stdout: librados-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:11.357 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:11.358 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:11.582 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs2 2024-06-06T03:08:11.714 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:40 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:11.794 INFO:teuthology.orchestra.run.smithi173.stdout:Package python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:11.837 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:11.838 INFO:teuthology.orchestra.run.smithi173.stdout:Nothing to do. 2024-06-06T03:08:11.838 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:11.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-fuse 2024-06-06T03:08:12.154 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:12.238 INFO:teuthology.orchestra.run.smithi153.stdout:Package libcephfs2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:12.280 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:12.281 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:12.281 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:12.367 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install libcephfs-devel 2024-06-06T03:08:12.486 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:41 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:12.608 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:12.609 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-06-06T03:08:12.609 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:12.609 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:12.609 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 82 k 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 82 k 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 222 k 2024-06-06T03:08:12.610 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:12.938 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:41 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:12.958 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-fuse-17.2.7-992.g6ee68927.el9.x86_64.rpm 235 kB/s | 82 kB 00:00 2024-06-06T03:08:12.958 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:12.959 INFO:teuthology.orchestra.run.smithi173.stdout:Total 234 kB/s | 82 kB 00:00 2024-06-06T03:08:12.959 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:12.965 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:12.966 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:13.027 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:13.028 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:13.062 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:13.063 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-06-06T03:08:13.063 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:13.063 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:13.063 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel x86_64 2:17.2.7-992.g6ee68927.el9 ceph 25 k 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 25 k 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 110 k 2024-06-06T03:08:13.064 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:13.173 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:13.233 INFO:teuthology.orchestra.run.smithi153.stdout:libcephfs-devel-17.2.7-992.g6ee68927.el9.x86_64 148 kB/s | 25 kB 00:00 2024-06-06T03:08:13.234 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:13.234 INFO:teuthology.orchestra.run.smithi153.stdout:Total 147 kB/s | 25 kB 00:00 2024-06-06T03:08:13.234 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:13.237 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:13.238 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:13.239 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:13.239 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:13.270 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:13.318 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:13.429 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:13.639 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:13.804 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:13.895 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-devel-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:14.032 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:14.142 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-mirror 2024-06-06T03:08:14.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librados2 2024-06-06T03:08:14.722 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:43 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:14.785 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:43 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:14.848 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:14.848 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout: Package Arch Version Repo Size 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:08:14.849 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:14.850 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 2.9 M 2024-06-06T03:08:14.850 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 12 M 2024-06-06T03:08:14.850 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:14.866 INFO:teuthology.orchestra.run.smithi153.stdout:Package librados2-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:14.907 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:14.908 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:14.909 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:14.973 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install librbd1 2024-06-06T03:08:15.236 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-mirror-17.2.7-992.g6ee68927.el9.x86_64.rpm 7.6 MB/s | 2.9 MB 00:00 2024-06-06T03:08:15.237 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:15.237 INFO:teuthology.orchestra.run.smithi173.stdout:Total 7.6 MB/s | 2.9 MB 00:00 2024-06-06T03:08:15.237 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:15.246 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:15.246 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:15.326 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:15.326 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:15.545 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:44 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:15.626 INFO:teuthology.orchestra.run.smithi153.stdout:Package librbd1-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:15.667 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:15.668 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:15.668 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:15.731 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rados 2024-06-06T03:08:15.759 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:15.829 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T03:08:15.853 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:16.301 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:45 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:16.383 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rados-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:16.426 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:16.426 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:16.426 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:16.489 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rgw 2024-06-06T03:08:16.968 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:16.969 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:16.969 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:16.969 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:16.969 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:16.969 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:17.060 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:17.140 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rgw-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:17.181 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:17.182 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:17.182 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:17.232 DEBUG:teuthology.orchestra.run.smithi173:> sudo yum -y install rbd-nbd 2024-06-06T03:08:17.245 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-cephfs 2024-06-06T03:08:17.813 INFO:teuthology.orchestra.run.smithi173.stdout:Last metadata expiration check: 0:01:46 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:17.816 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:46 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:17.897 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-cephfs-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:17.936 INFO:teuthology.orchestra.run.smithi173.stdout:Dependencies resolved. 2024-06-06T03:08:17.937 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: Package Architecture Version Repository Size 2024-06-06T03:08:17.937 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:17.937 INFO:teuthology.orchestra.run.smithi173.stdout:Installing: 2024-06-06T03:08:17.937 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 165 k 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction Summary 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:================================================================================ 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:Install 1 Package 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:Total download size: 165 k 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:Installed size: 477 k 2024-06-06T03:08:17.938 INFO:teuthology.orchestra.run.smithi173.stdout:Downloading Packages: 2024-06-06T03:08:17.939 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:17.940 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:17.940 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:18.000 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install python3-rbd 2024-06-06T03:08:18.297 INFO:teuthology.orchestra.run.smithi173.stdout:rbd-nbd-17.2.7-992.g6ee68927.el9.x86_64.rpm 459 kB/s | 165 kB 00:00 2024-06-06T03:08:18.298 INFO:teuthology.orchestra.run.smithi173.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:18.298 INFO:teuthology.orchestra.run.smithi173.stdout:Total 457 kB/s | 165 kB 00:00 2024-06-06T03:08:18.298 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction check 2024-06-06T03:08:18.306 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction check succeeded. 2024-06-06T03:08:18.306 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction test 2024-06-06T03:08:18.369 INFO:teuthology.orchestra.run.smithi173.stdout:Transaction test succeeded. 2024-06-06T03:08:18.369 INFO:teuthology.orchestra.run.smithi173.stdout:Running transaction 2024-06-06T03:08:18.515 INFO:teuthology.orchestra.run.smithi173.stdout: Preparing : 1/1 2024-06-06T03:08:18.570 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:47 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:18.611 INFO:teuthology.orchestra.run.smithi173.stdout: Installing : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:18.651 INFO:teuthology.orchestra.run.smithi153.stdout:Package python3-rbd-2:17.2.7-992.g6ee68927.el9.x86_64 is already installed. 2024-06-06T03:08:18.693 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:18.693 INFO:teuthology.orchestra.run.smithi153.stdout:Nothing to do. 2024-06-06T03:08:18.694 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:18.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-fuse 2024-06-06T03:08:19.024 INFO:teuthology.orchestra.run.smithi173.stdout: Running scriptlet: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: Verifying : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:19.303 INFO:teuthology.orchestra.run.smithi173.stdout:Installed: 2024-06-06T03:08:19.303 INFO:teuthology.orchestra.run.smithi173.stdout: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:19.304 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:08:19.304 INFO:teuthology.orchestra.run.smithi173.stdout:Complete! 2024-06-06T03:08:19.332 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:48 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:19.456 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse x86_64 2:17.2.7-992.g6ee68927.el9 ceph 82 k 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:19.457 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:08:19.458 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:19.458 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 82 k 2024-06-06T03:08:19.458 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 222 k 2024-06-06T03:08:19.458 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:19.552 DEBUG:teuthology.parallel:result is None 2024-06-06T03:08:19.679 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-fuse-17.2.7-992.g6ee68927.el9.x86_64.rpm 372 kB/s | 82 kB 00:00 2024-06-06T03:08:19.680 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:19.680 INFO:teuthology.orchestra.run.smithi153.stdout:Total 369 kB/s | 82 kB 00:00 2024-06-06T03:08:19.680 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:19.686 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:19.686 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:19.748 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:19.749 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:19.892 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:20.000 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:20.390 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-fuse-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:20.643 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:20.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-mirror 2024-06-06T03:08:21.455 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:50 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:21.581 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:21.582 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:21.582 INFO:teuthology.orchestra.run.smithi153.stdout: Package Arch Version Repo Size 2024-06-06T03:08:21.582 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror x86_64 2:17.2.7-992.g6ee68927.el9 ceph 2.9 M 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 2.9 M 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 12 M 2024-06-06T03:08:21.583 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:22.122 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-mirror-17.2.7-992.g6ee68927.el9.x86_64.rpm 5.5 MB/s | 2.9 MB 00:00 2024-06-06T03:08:22.123 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:22.123 INFO:teuthology.orchestra.run.smithi153.stdout:Total 5.4 MB/s | 2.9 MB 00:00 2024-06-06T03:08:22.124 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:22.132 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:22.133 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:22.210 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:22.211 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:22.636 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:22.698 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-06T03:08:22.723 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:23.814 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:23.815 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-mirror-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:23.815 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:23.815 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:24.042 DEBUG:teuthology.orchestra.run.smithi153:> sudo yum -y install rbd-nbd 2024-06-06T03:08:24.629 INFO:teuthology.orchestra.run.smithi153.stdout:Last metadata expiration check: 0:01:53 ago on Thu 06 Jun 2024 03:06:31 AM UTC. 2024-06-06T03:08:24.751 INFO:teuthology.orchestra.run.smithi153.stdout:Dependencies resolved. 2024-06-06T03:08:24.751 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout: Package Architecture Version Repository Size 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout:Installing: 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd x86_64 2:17.2.7-992.g6ee68927.el9 ceph 165 k 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction Summary 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout:================================================================================ 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout:Install 1 Package 2024-06-06T03:08:24.752 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:24.753 INFO:teuthology.orchestra.run.smithi153.stdout:Total download size: 165 k 2024-06-06T03:08:24.753 INFO:teuthology.orchestra.run.smithi153.stdout:Installed size: 477 k 2024-06-06T03:08:24.753 INFO:teuthology.orchestra.run.smithi153.stdout:Downloading Packages: 2024-06-06T03:08:24.983 INFO:teuthology.orchestra.run.smithi153.stdout:rbd-nbd-17.2.7-992.g6ee68927.el9.x86_64.rpm 717 kB/s | 165 kB 00:00 2024-06-06T03:08:24.984 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:08:24.984 INFO:teuthology.orchestra.run.smithi153.stdout:Total 713 kB/s | 165 kB 00:00 2024-06-06T03:08:24.984 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction check 2024-06-06T03:08:24.991 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction check succeeded. 2024-06-06T03:08:24.991 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction test 2024-06-06T03:08:25.054 INFO:teuthology.orchestra.run.smithi153.stdout:Transaction test succeeded. 2024-06-06T03:08:25.054 INFO:teuthology.orchestra.run.smithi153.stdout:Running transaction 2024-06-06T03:08:25.205 INFO:teuthology.orchestra.run.smithi153.stdout: Preparing : 1/1 2024-06-06T03:08:25.301 INFO:teuthology.orchestra.run.smithi153.stdout: Installing : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:25.751 INFO:teuthology.orchestra.run.smithi153.stdout: Running scriptlet: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout: Verifying : rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 1/1 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout:Installed: 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout: rbd-nbd-2:17.2.7-992.g6ee68927.el9.x86_64 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:08:26.018 INFO:teuthology.orchestra.run.smithi153.stdout:Complete! 2024-06-06T03:08:26.241 DEBUG:teuthology.parallel:result is None 2024-06-06T03:08:26.242 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:08:26.242 INFO:teuthology.packaging:ref: None 2024-06-06T03:08:26.242 INFO:teuthology.packaging:tag: None 2024-06-06T03:08:26.242 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:08:26.242 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:08:26.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T03:08:26.445 DEBUG:teuthology.orchestra.run.smithi153:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T03:08:26.466 INFO:teuthology.orchestra.run.smithi153.stdout:17.2.7-992.g6ee68927.el9 2024-06-06T03:08:26.466 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-992.g6ee68927.el9 2024-06-06T03:08:26.467 INFO:teuthology.task.install:The correct ceph version 17.2.7-992.g6ee68927 is installed. 2024-06-06T03:08:26.468 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-06T03:08:26.468 INFO:teuthology.packaging:ref: None 2024-06-06T03:08:26.468 INFO:teuthology.packaging:tag: None 2024-06-06T03:08:26.468 INFO:teuthology.packaging:branch: quincy 2024-06-06T03:08:26.469 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-06T03:08:26.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-06T03:08:26.759 DEBUG:teuthology.orchestra.run.smithi173:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-06T03:08:26.786 INFO:teuthology.orchestra.run.smithi173.stdout:17.2.7-992.g6ee68927.el9 2024-06-06T03:08:26.786 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-992.g6ee68927.el9 2024-06-06T03:08:26.786 INFO:teuthology.task.install:The correct ceph version 17.2.7-992.g6ee68927 is installed. 2024-06-06T03:08:26.788 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-06T03:08:26.788 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:08:26.788 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T03:08:26.814 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:08:26.814 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-06T03:08:26.856 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-06T03:08:26.856 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:08:26.856 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T03:08:26.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T03:08:26.946 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:08:26.946 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/daemon-helper 2024-06-06T03:08:26.973 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-06T03:08:27.038 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-06T03:08:27.038 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:08:27.038 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T03:08:27.063 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T03:08:27.128 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:08:27.128 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-06T03:08:27.154 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-06T03:08:27.218 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-06T03:08:27.218 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:08:27.218 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T03:08:27.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T03:08:27.307 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:08:27.307 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/usr/bin/stdin-killer 2024-06-06T03:08:27.334 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-06T03:08:27.399 INFO:teuthology.run_tasks:Running task print... 2024-06-06T03:08:27.425 INFO:teuthology.task.print:**** done install task... 2024-06-06T03:08:27.425 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-06T03:08:27.586 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-06T03:08:27.586 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-06T03:08:27.586 INFO:tasks.cephadm:Cluster fsid is 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:08:27.587 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-06T03:08:27.587 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-06T03:08:27.587 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi153': '172.21.15.153', 'mon.smithi173': '172.21.15.173'} 2024-06-06T03:08:27.587 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-06T03:08:27.587 DEBUG:teuthology.orchestra.run.smithi153:> sudo hostname $(hostname -s) 2024-06-06T03:08:27.612 DEBUG:teuthology.orchestra.run.smithi173:> sudo hostname $(hostname -s) 2024-06-06T03:08:27.636 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-06T03:08:27.637 DEBUG:teuthology.orchestra.run.smithi153:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:27.943 INFO:teuthology.orchestra.run.smithi153.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 6 03:08 /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:27.944 DEBUG:teuthology.orchestra.run.smithi173:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:28.155 INFO:teuthology.orchestra.run.smithi173.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 6 03:08 /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:28.155 DEBUG:teuthology.orchestra.run.smithi153:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:28.174 DEBUG:teuthology.orchestra.run.smithi173:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-06T03:08:28.252 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-06T03:08:28.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-06T03:08:28.258 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-06T03:08:29.097 INFO:teuthology.orchestra.run.smithi173.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T03:08:29.175 INFO:teuthology.orchestra.run.smithi153.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T03:09:04.793 INFO:teuthology.orchestra.run.smithi173.stdout:{ 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout: "ceph_version": "ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable)", 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout: "image_id": "7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a", 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout: "repo_digests": [ 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a" 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout: ] 2024-06-06T03:09:04.794 INFO:teuthology.orchestra.run.smithi173.stdout:} 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout: "ceph_version": "ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable)", 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout: "image_id": "7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a", 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout: "repo_digests": [ 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a" 2024-06-06T03:09:05.934 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-06-06T03:09:05.935 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-06-06T03:09:05.953 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph 2024-06-06T03:09:06.003 DEBUG:teuthology.orchestra.run.smithi173:> sudo mkdir -p /etc/ceph 2024-06-06T03:09:06.037 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /etc/ceph 2024-06-06T03:09:06.062 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 777 /etc/ceph 2024-06-06T03:09:06.107 INFO:tasks.cephadm:Writing seed config... 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-06T03:09:06.108 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-06T03:09:06.109 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-06T03:09:06.110 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-06-06T03:09:06.111 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-06T03:09:06.112 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:09:06.112 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-06T03:09:06.127 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 0b80117e-23b2-11ef-bc9b-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-06T03:09:06.127 DEBUG:teuthology.orchestra.run.smithi153:mon.smithi153> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service 2024-06-06T03:09:06.170 INFO:tasks.cephadm:Bootstrapping... 2024-06-06T03:09:06.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.153 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:09:06.374 INFO:teuthology.orchestra.run.smithi153.stdout:-------------------------------------------------------------------------------- 2024-06-06T03:09:06.374 INFO:teuthology.orchestra.run.smithi153.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', '0b80117e-23b2-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.153', '--skip-admin-label'] 2024-06-06T03:09:06.399 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 5.0.2 2024-06-06T03:09:06.400 INFO:teuthology.orchestra.run.smithi153.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-06T03:09:06.400 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying podman|docker is present... 2024-06-06T03:09:06.423 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 5.0.2 2024-06-06T03:09:06.423 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying lvm2 is present... 2024-06-06T03:09:06.423 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying time synchronization is in place... 2024-06-06T03:09:06.431 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T03:09:06.431 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T03:09:06.438 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T03:09:06.438 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:09:06.445 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-06-06T03:09:06.452 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-06-06T03:09:06.452 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-06-06T03:09:06.452 INFO:teuthology.orchestra.run.smithi153.stdout:Repeating the final host check... 2024-06-06T03:09:06.475 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 5.0.2 2024-06-06T03:09:06.475 INFO:teuthology.orchestra.run.smithi153.stdout:podman (/bin/podman) version 5.0.2 is present 2024-06-06T03:09:06.475 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl is present 2024-06-06T03:09:06.475 INFO:teuthology.orchestra.run.smithi153.stdout:lvcreate is present 2024-06-06T03:09:06.482 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-06T03:09:06.482 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-06T03:09:06.488 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-06T03:09:06.488 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:09:06.495 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout enabled 2024-06-06T03:09:06.502 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout active 2024-06-06T03:09:06.502 INFO:teuthology.orchestra.run.smithi153.stdout:Unit chronyd.service is enabled and running 2024-06-06T03:09:06.502 INFO:teuthology.orchestra.run.smithi153.stdout:Host looks OK 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Cluster fsid: 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Acquiring lock 140684469699008 on /run/cephadm/0b80117e-23b2-11ef-bc9b-c7b262605968.lock 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Lock 140684469699008 acquired on /run/cephadm/0b80117e-23b2-11ef-bc9b-c7b262605968.lock 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 3300 ... 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying IP 172.21.15.153 port 6789 ... 2024-06-06T03:09:06.503 INFO:teuthology.orchestra.run.smithi153.stdout:Base mon IP(s) is [172.21.15.153:3300, 172.21.15.153:6789], mon addrv is [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-06-06T03:09:06.507 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.153 metric 100 2024-06-06T03:09:06.507 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.153 metric 100 2024-06-06T03:09:06.510 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-06T03:09:06.510 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-06T03:09:06.510 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6deb/64 scope link 2024-06-06T03:09:06.513 INFO:teuthology.orchestra.run.smithi153.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-06T03:09:06.514 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-06-06T03:09:06.514 INFO:teuthology.orchestra.run.smithi153.stdout:Mon IP `172.21.15.153` is in CIDR network `172.21.0.0/20` 2024-06-06T03:09:06.514 INFO:teuthology.orchestra.run.smithi153.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-06T03:09:06.514 INFO:teuthology.orchestra.run.smithi153.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-06T03:09:06.515 INFO:teuthology.orchestra.run.smithi153.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stdout 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Getting image source signatures 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:63e3eeadb2c456ff44649abe69523acc2aac9d8a9fe03158716a851e0cecadea 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying blob sha256:36850128dbaa55a7fd38bf12640a70ab57c121fdc76af02f72b423e059979691 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Copying config sha256:7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a 2024-06-06T03:09:07.034 INFO:teuthology.orchestra.run.smithi153.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-06T03:09:07.938 INFO:teuthology.orchestra.run.smithi153.stdout:ceph: stdout ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable) 2024-06-06T03:09:07.938 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph version: ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable) 2024-06-06T03:09:07.938 INFO:teuthology.orchestra.run.smithi153.stdout:Extracting ceph user uid/gid from container image... 2024-06-06T03:09:08.785 INFO:teuthology.orchestra.run.smithi153.stdout:stat: stdout 167 167 2024-06-06T03:09:08.785 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial keys... 2024-06-06T03:09:09.666 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBVKGFmMmT2BRAAax10u+8KgqSC5q5TVOj8Ww== 2024-06-06T03:09:10.638 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBWKGFmS2VjAhAAKRqH+8m5vs7ewsEPz9DJNg== 2024-06-06T03:09:11.518 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph-authtool: stdout AQBWKGFm1K9VOBAAAqwJUbMNZUDcSlN+I0GlAw== 2024-06-06T03:09:11.519 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial monmap... 2024-06-06T03:09:12.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T03:09:12.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-06T03:09:12.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:12.433 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool for smithi153 [v2:172.21.15.153:3300,v1:172.21.15.153:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = octopus 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: set fsid to 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:09:12.434 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mon... 2024-06-06T03:09:13.355 INFO:teuthology.orchestra.run.smithi153.stdout:create mon.smithi153 on 2024-06-06T03:09:13.576 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-06T03:09:13.722 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-06T03:09:13.873 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target. 2024-06-06T03:09:13.874 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target. 2024-06-06T03:09:14.047 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153 2024-06-06T03:09:14.047 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service: Unit ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service not loaded. 2024-06-06T03:09:14.217 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target.wants/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service → /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@.service. 2024-06-06T03:09:14.753 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T03:09:14.754 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-06-06T03:09:14.761 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T03:09:14.761 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:09:14.761 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-06-06T03:09:14.761 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T03:09:14.762 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon to start... 2024-06-06T03:09:14.762 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mon... 2024-06-06T03:09:15.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout cluster: 2024-06-06T03:09:15.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout id: 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:15.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-06T03:09:15.746 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout services: 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi153 (age 0.524695s) 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout data: 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout pgs: 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe1a1a23640 1 Processor -- start 2024-06-06T03:09:15.747 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe1a1a23640 1 -- start start 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe1a1a23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe1a1a23640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1026e0 con 0x7fe19c101d90 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe19affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe19affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53840/0 (socket says 172.21.15.153:53840) 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.135+0000 7fe19affd640 1 -- 172.21.15.153:0/2045748052 learned_addr learned my addr 172.21.15.153:0/2045748052 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.136+0000 7fe19affd640 1 -- 172.21.15.153:0/2045748052 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c102820 con 0x7fe19c101d90 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.137+0000 7fe19affd640 1 --2- 172.21.15.153:0/2045748052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009840 tx=0x7fe18c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad2ec75affb2677 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe199ffb640 1 -- 172.21.15.153:0/2045748052 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe18c032440 con 0x7fe19c101d90 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe199ffb640 1 -- 172.21.15.153:0/2045748052 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c0325a0 con 0x7fe19c101d90 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe199ffb640 1 -- 172.21.15.153:0/2045748052 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe18c0328d0 con 0x7fe19c101d90 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 -- 172.21.15.153:0/2045748052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 msgr2=0x7fe19c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 --2- 172.21.15.153:0/2045748052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe18c009840 tx=0x7fe18c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:15.748 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 -- 172.21.15.153:0/2045748052 shutdown_connections 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 --2- 172.21.15.153:0/2045748052 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c1021a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 -- 172.21.15.153:0/2045748052 >> 172.21.15.153:0/2045748052 conn(0x7fe19c0fd5b0 msgr2=0x7fe19c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 -- 172.21.15.153:0/2045748052 shutdown_connections 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.138+0000 7fe1a1a23640 1 -- 172.21.15.153:0/2045748052 wait complete. 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe1a1a23640 1 Processor -- start 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe1a1a23640 1 -- start start 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe1a1a23640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe1a1a23640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c110b70 con 0x7fe19c101d90 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe19affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.139+0000 7fe19affd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53844/0 (socket says 172.21.15.153:53844) 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe19affd640 1 -- 172.21.15.153:0/3017790848 learned_addr learned my addr 172.21.15.153:0/3017790848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:15.749 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe19affd640 1 -- 172.21.15.153:0/3017790848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe18c009450 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe19affd640 1 --2- 172.21.15.153:0/3017790848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe18c02a700 tx=0x7fe18c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe18c031be0 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18c031d40 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe18c03b5a0 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c110d70 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c111180 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.140+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fe18c039040 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.141+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fe18c041030 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.142+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c04b480 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.145+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fe18c037070 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.208+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fe19c103030 con 0x7fe19c101d90 2024-06-06T03:09:15.750 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.208+0000 7fe1837fe640 1 -- 172.21.15.153:0/3017790848 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7fe18c054070 con 0x7fe19c101d90 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 msgr2=0x7fe19c110630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 --2- 172.21.15.153:0/3017790848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fe18c02a700 tx=0x7fe18c032000 comp rx=0 tx=0).stop 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 shutdown_connections 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 --2- 172.21.15.153:0/3017790848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe19c101d90 0x7fe19c110630 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 >> 172.21.15.153:0/3017790848 conn(0x7fe19c0fd5b0 msgr2=0x7fe19c107220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 shutdown_connections 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:15.210+0000 7fe1a1a23640 1 -- 172.21.15.153:0/3017790848 wait complete. 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:mon is available 2024-06-06T03:09:15.751 INFO:teuthology.orchestra.run.smithi153.stdout:Assimilating anything we can from ceph.conf... 2024-06-06T03:09:16.802 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:16 smithi153 ceph-mon[28932]: from='client.? 172.21.15.153:0/3991285777' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-06T03:09:16.809 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.153:3300,v1:172.21.15.153:6789] 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:16.810 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cb2265640 1 Processor -- start 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cb2265640 1 -- start start 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cb2265640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cb2265640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac100d10 con 0x7f6cac1003c0 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53854/0 (socket says 172.21.15.153:53854) 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.156+0000 7f6cabfff640 1 -- 172.21.15.153:0/3653155942 learned_addr learned my addr 172.21.15.153:0/3653155942 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.157+0000 7f6cabfff640 1 -- 172.21.15.153:0/3653155942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cac100e50 con 0x7f6cac1003c0 2024-06-06T03:09:16.811 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.157+0000 7f6cabfff640 1 --2- 172.21.15.153:0/3653155942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009840 tx=0x7f6c9c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f2b192d1058f7df server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.157+0000 7f6caaffd640 1 -- 172.21.15.153:0/3653155942 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c9c032440 con 0x7f6cac1003c0 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.157+0000 7f6caaffd640 1 -- 172.21.15.153:0/3653155942 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c0325a0 con 0x7f6cac1003c0 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.157+0000 7f6caaffd640 1 -- 172.21.15.153:0/3653155942 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c9c0328d0 con 0x7f6cac1003c0 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 -- 172.21.15.153:0/3653155942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 msgr2=0x7f6cac1007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 --2- 172.21.15.153:0/3653155942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c009840 tx=0x7f6c9c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 -- 172.21.15.153:0/3653155942 shutdown_connections 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 --2- 172.21.15.153:0/3653155942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac1007d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 -- 172.21.15.153:0/3653155942 >> 172.21.15.153:0/3653155942 conn(0x7f6cac0fbc20 msgr2=0x7f6cac0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 -- 172.21.15.153:0/3653155942 shutdown_connections 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 -- 172.21.15.153:0/3653155942 wait complete. 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.158+0000 7f6cb2265640 1 Processor -- start 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cb2265640 1 -- start start 2024-06-06T03:09:16.812 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cb2265640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cb2265640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cac110850 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cabfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53864/0 (socket says 172.21.15.153:53864) 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cabfff640 1 -- 172.21.15.153:0/3991285777 learned_addr learned my addr 172.21.15.153:0/3991285777 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cabfff640 1 -- 172.21.15.153:0/3991285777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c9c009450 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.159+0000 7f6cabfff640 1 --2- 172.21.15.153:0/3991285777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c02a700 tx=0x7f6c9c032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c9c031be0 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c9c031d40 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cac110a50 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c9c03b5a0 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cac110e60 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f6c9c039040 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.160+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f6c9c041030 con 0x7f6cac1003c0 2024-06-06T03:09:16.813 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.161+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cac04b480 con 0x7f6cac1003c0 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.163+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f6c9c037070 con 0x7f6cac1003c0 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.221+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f6cac1115e0 con 0x7f6cac1003c0 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6c9c046d00 con 0x7f6cac1003c0 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6ca8ff9640 1 -- 172.21.15.153:0/3991285777 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f6c9c04eca0 con 0x7f6cac1003c0 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 msgr2=0x7f6cac110310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6cb2265640 1 --2- 172.21.15.153:0/3991285777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f6c9c02a700 tx=0x7f6c9c032000 comp rx=0 tx=0).stop 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 shutdown_connections 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6cb2265640 1 --2- 172.21.15.153:0/3991285777 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6cac1003c0 0x7f6cac110310 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.225+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 >> 172.21.15.153:0/3991285777 conn(0x7f6cac0fbc20 msgr2=0x7f6cac104960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:16.814 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.226+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 shutdown_connections 2024-06-06T03:09:16.815 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:16.226+0000 7f6cb2265640 1 -- 172.21.15.153:0/3991285777 wait complete. 2024-06-06T03:09:16.815 INFO:teuthology.orchestra.run.smithi153.stdout:Generating new minimal ceph.conf... 2024-06-06T03:09:17.791 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:17 smithi153 ceph-mon[28932]: from='client.? 172.21.15.153:0/273745094' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:09:17.797 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8cca7f640 1 Processor -- start 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8cca7f640 1 -- start start 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8cca7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8cca7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8075f30 con 0x7fa8c8075b20 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53874/0 (socket says 172.21.15.153:53874) 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8c659c640 1 -- 172.21.15.153:0/2314440992 learned_addr learned my addr 172.21.15.153:0/2314440992 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.170+0000 7fa8c659c640 1 -- 172.21.15.153:0/2314440992 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8102830 con 0x7fa8c8075b20 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.171+0000 7fa8c659c640 1 --2- 172.21.15.153:0/2314440992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009840 tx=0x7fa8b80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d253226d575ca5 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.171+0000 7fa8c5d9b640 1 -- 172.21.15.153:0/2314440992 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8032440 con 0x7fa8c8075b20 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.171+0000 7fa8c5d9b640 1 -- 172.21.15.153:0/2314440992 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa8b8032a30 con 0x7fa8c8075b20 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.171+0000 7fa8c5d9b640 1 -- 172.21.15.153:0/2314440992 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b8031ad0 con 0x7fa8c8075b20 2024-06-06T03:09:17.798 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 -- 172.21.15.153:0/2314440992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 msgr2=0x7fa8c8106e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 --2- 172.21.15.153:0/2314440992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009840 tx=0x7fa8b80091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 -- 172.21.15.153:0/2314440992 shutdown_connections 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 --2- 172.21.15.153:0/2314440992 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8106e40 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 -- 172.21.15.153:0/2314440992 >> 172.21.15.153:0/2314440992 conn(0x7fa8c80fc4d0 msgr2=0x7fa8c80fe8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 -- 172.21.15.153:0/2314440992 shutdown_connections 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.172+0000 7fa8cca7f640 1 -- 172.21.15.153:0/2314440992 wait complete. 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8cca7f640 1 Processor -- start 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8cca7f640 1 -- start start 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8cca7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8cca7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c81131c0 con 0x7fa8c8075b20 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:17.799 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8c659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53890/0 (socket says 172.21.15.153:53890) 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8c659c640 1 -- 172.21.15.153:0/273745094 learned_addr learned my addr 172.21.15.153:0/273745094 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8c659c640 1 -- 172.21.15.153:0/273745094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009450 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8c659c640 1 --2- 172.21.15.153:0/273745094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8006f00 tx=0x7fa8b8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b803d450 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa8b803da40 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c8113420 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.173+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8b803bb10 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.174+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c8113f50 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.174+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fa8b8039070 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.174+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa8b8046930 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.175+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c805ead0 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.177+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fa8b8041020 con 0x7fa8c8075b20 2024-06-06T03:09:17.800 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.235+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa8c8107f70 con 0x7fa8c8075b20 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.235+0000 7fa8beffd640 1 -- 172.21.15.153:0/273745094 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fa8b8037070 con 0x7fa8c8075b20 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 msgr2=0x7fa8c8112c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 --2- 172.21.15.153:0/273745094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8006f00 tx=0x7fa8b8008040 comp rx=0 tx=0).stop 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 shutdown_connections 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 --2- 172.21.15.153:0/273745094 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8c8075b20 0x7fa8c8112c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 >> 172.21.15.153:0/273745094 conn(0x7fa8c80fc4d0 msgr2=0x7fa8c80fd080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 shutdown_connections 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:17.236+0000 7fa8cca7f640 1 -- 172.21.15.153:0/273745094 wait complete. 2024-06-06T03:09:17.801 INFO:teuthology.orchestra.run.smithi153.stdout:Restarting the monitor... 2024-06-06T03:09:18.061 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:17 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:09:18.453 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153[28928]: 2024-06-06T03:09:18.061+0000 7fd55c84b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:09:18.453 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153[28928]: 2024-06-06T03:09:18.061+0000 7fd55c84b640 -1 mon.smithi153@0(leader) e1 *** Got Signal Terminated *** 2024-06-06T03:09:18.453 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 podman[29175]: 2024-06-06 03:09:18.225225178 +0000 UTC m=+0.265375584 container died 2f8c4f061cf7de3a4df11b83e1b4d5d883462be07b86f504acf54fbef9a810f8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-06T03:09:18.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 podman[29175]: 2024-06-06 03:09:18.552941331 +0000 UTC m=+0.593091735 container cleanup 2f8c4f061cf7de3a4df11b83e1b4d5d883462be07b86f504acf54fbef9a810f8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, ceph=True, org.label-schema.license=GPLv2) 2024-06-06T03:09:18.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 bash[29175]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153 2024-06-06T03:09:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 podman[29188]: 2024-06-06 03:09:18.703887544 +0000 UTC m=+0.478993799 container remove 2f8c4f061cf7de3a4df11b83e1b4d5d883462be07b86f504acf54fbef9a810f8 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-06-06T03:09:19.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 systemd[1]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service: Deactivated successfully. 2024-06-06T03:09:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968. 2024-06-06T03:09:19.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:18 smithi153 systemd[1]: Starting Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:09:19.397 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 podman[29281]: 2024-06-06 03:09:19.129632474 +0000 UTC m=+0.106192046 container create 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 podman[29281]: 2024-06-06 03:09:19.04058593 +0000 UTC m=+0.017145470 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph:quincy 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 podman[29281]: 2024-06-06 03:09:19.323332168 +0000 UTC m=+0.299891694 container init 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 podman[29281]: 2024-06-06 03:09:19.326293246 +0000 UTC m=+0.302852774 container start 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: set uid:gid to 167:167 (ceph:ceph) 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: ceph version 17.2.7-992-g6ee68927 (6ee689278780be842e5800190ee2e2bd3d5027f6) quincy (stable), process ceph-mon, pid 2 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: pidfile_write: ignore empty --pid-file 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: load: jerasure load: lrc 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: RocksDB version: 6.15.5 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Compile date Jun 5 2024 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: DB SUMMARY 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: DB Session ID: NMFGJORVW47MXYF8H9KX 2024-06-06T03:09:19.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: CURRENT file: CURRENT 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: IDENTITY file: IDENTITY 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi153/store.db dir, Total Num: 1, files: 000008.sst 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi153/store.db: 000010.log size: 98190 ; 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.error_if_exists: 0 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.create_if_missing: 0 2024-06-06T03:09:19.399 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.paranoid_checks: 1 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.env: 0x555c6327a440 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.fs: Posix File System 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.info_log: 0x555c64bef140 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_file_opening_threads: 16 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.statistics: (nil) 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.use_fsync: 0 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_log_file_size: 0 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-06T03:09:19.401 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.keep_log_file_num: 1000 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.recycle_log_file_num: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_fallocate: 1 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_mmap_reads: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_mmap_writes: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.use_direct_reads: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.create_missing_column_families: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.db_log_dir: 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi153/store.db 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-06T03:09:19.402 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.advise_random_on_open: 1 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.db_write_buffer_size: 0 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.write_buffer_manager: 0x555c64cae7e0 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.rate_limiter: (nil) 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-06T03:09:19.403 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.wal_recovery_mode: 2 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.enable_thread_tracking: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.enable_pipelined_write: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.unordered_write: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.row_cache: None 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.wal_filter: None 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_ingest_behind: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.preserve_deletes: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.two_write_queues: 0 2024-06-06T03:09:19.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.manual_wal_flush: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.atomic_flush: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.log_readahead_size: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.best_efforts_recovery: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.allow_data_in_errors: 0 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.db_host_id: __hostname__ 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_background_jobs: 2 2024-06-06T03:09:19.405 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_background_compactions: -1 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_subcompactions: 1 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_total_wal_size: 0 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_open_files: -1 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bytes_per_sync: 0 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-06T03:09:19.406 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_readahead_size: 0 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_background_flushes: -1 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Compression algorithms supported: 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kZSTD supported: 0 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kXpressCompression supported: 0 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kBZip2Compression supported: 0 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kLZ4Compression supported: 1 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kZlibCompression supported: 1 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kLZ4HCCompression supported: 1 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: kSnappyCompression supported: 1 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 2024-06-06T03:09:19.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.merge_operator: 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_filter: None 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_filter_factory: None 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.sst_partitioner_factory: None 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555c64b97a68) 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks: 1 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: pin_top_level_index_and_filter: 1 2024-06-06T03:09:19.408 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_type: 0 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_index_type: 0 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_shortening: 1 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: hash_index_allow_collision: 1 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: checksum: 1 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: no_block_cache: 0 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache: 0x555c64bcd1f0 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_name: BinnedLRUCache 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_options: 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: capacity : 536870912 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: num_shard_bits : 4 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: strict_capacity_limit : 0 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: high_pri_pool_ratio: 0.000 2024-06-06T03:09:19.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_cache_compressed: (nil) 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: persistent_cache: (nil) 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size: 4096 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_size_deviation: 10 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_restart_interval: 16 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: index_block_restart_interval: 1 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: metadata_block_size: 4096 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: partition_filters: 0 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: use_delta_encoding: 1 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: whole_key_filtering: 1 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: verify_compression: 0 2024-06-06T03:09:19.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: read_amp_bytes_per_bit: 0 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: format_version: 4 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_index_compression: 1 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: block_align: 0 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.write_buffer_size: 33554432 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_write_buffer_number: 2 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression: NoCompression 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression: Disabled 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.prefix_extractor: nullptr 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.num_levels: 7 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-06T03:09:19.411 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.level: 32767 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.strategy: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-06T03:09:19.412 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compression_opts.enabled: false 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.target_file_size_base: 67108864 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-06T03:09:19.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.arena_block_size: 4194304 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.disable_auto_compactions: 0 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-06T03:09:19.414 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.table_properties_collectors: 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.inplace_update_support: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.bloom_locality: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.max_successive_merges: 0 2024-06-06T03:09:19.415 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.paranoid_file_checks: 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.force_consistency_checks: 1 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.report_bg_io_stats: 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.ttl: 2592000 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.enable_blob_files: false 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.min_blob_size: 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.blob_file_size: 268435456 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi153/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-06T03:09:19.416 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717643359348552, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717643359349429, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 93396, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 91508, "index_size": 248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13576, "raw_average_key_size": 49, "raw_value_size": 84072, "raw_average_value_size": 307, "num_data_blocks": 11, "num_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1717643359, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3ba58515-2830-4fac-bb1f-bde585768881", "db_session_id": "NMFGJORVW47MXYF8H9KX"}} 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: EVENT_LOG_v1 {"time_micros": 1717643359349807, "job": 1, "event": "recovery_finished"} 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi153/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x555c64c00700 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: DB pointer 0x555c64bd0000 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-06T03:09:19.417 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** DB Stats ** 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 93.00 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.418 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 93.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 28.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T03:09:19.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 28.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: L0 2/0 93.00 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Sum 2/0 93.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** Compaction Stats [default] ** 2024-06-06T03:09:19.420 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.4 0.00 0.00 1 0.001 0 0 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Cumulative compaction: 0.00 GB write, 27.72 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.421 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: starting mon.smithi153 rank 0 at public addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] at bind addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi153 fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???) e1 preinit fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).mds e1 new map 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).mds e1 print_map 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: e1 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: legacy client fscid: -1 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: No filesystems configured 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T03:09:19.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T03:09:19.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-06T03:09:19.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-06T03:09:19.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 bash[29281]: 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 2024-06-06T03:09:19.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 systemd[1]: Started Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968. 2024-06-06T03:09:19.423 INFO:teuthology.orchestra.run.smithi153.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-06T03:09:19.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mon.smithi153 is new leader, mons smithi153 in quorum (ranks 0) 2024-06-06T03:09:19.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: monmap e1: 1 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T03:09:19.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: fsmap 2024-06-06T03:09:19.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: osdmap e1: 0 total, 0 up, 0 in 2024-06-06T03:09:19.708 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:19 smithi153 ceph-mon[29300]: mgrmap e1: no daemons active 2024-06-06T03:09:20.331 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf951be640 1 Processor -- start 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf951be640 1 -- start start 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf951be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf951be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90102600 con 0x7fbf90101cb0 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf8ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf8ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53904/0 (socket says 172.21.15.153:53904) 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.787+0000 7fbf8ed9d640 1 -- 172.21.15.153:0/3016174224 learned_addr learned my addr 172.21.15.153:0/3016174224 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.788+0000 7fbf8ed9d640 1 -- 172.21.15.153:0/3016174224 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf90102740 con 0x7fbf90101cb0 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.789+0000 7fbf8ed9d640 1 --2- 172.21.15.153:0/3016174224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009840 tx=0x7fbf800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fc757d6d1aecf8 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.789+0000 7fbf8dd9b640 1 -- 172.21.15.153:0/3016174224 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf80032440 con 0x7fbf90101cb0 2024-06-06T03:09:20.336 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.789+0000 7fbf8dd9b640 1 -- 172.21.15.153:0/3016174224 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbf80032a30 con 0x7fbf90101cb0 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.789+0000 7fbf8dd9b640 1 -- 172.21.15.153:0/3016174224 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf80031ad0 con 0x7fbf90101cb0 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 -- 172.21.15.153:0/3016174224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 msgr2=0x7fbf901020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 --2- 172.21.15.153:0/3016174224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009840 tx=0x7fbf800091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 -- 172.21.15.153:0/3016174224 shutdown_connections 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 --2- 172.21.15.153:0/3016174224 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf901020c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 -- 172.21.15.153:0/3016174224 >> 172.21.15.153:0/3016174224 conn(0x7fbf900fd490 msgr2=0x7fbf900ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 -- 172.21.15.153:0/3016174224 shutdown_connections 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.790+0000 7fbf951be640 1 -- 172.21.15.153:0/3016174224 wait complete. 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf951be640 1 Processor -- start 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf951be640 1 -- start start 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf951be640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:20.337 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf951be640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf901113e0 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf8ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf8ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53914/0 (socket says 172.21.15.153:53914) 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf8ed9d640 1 -- 172.21.15.153:0/3792071940 learned_addr learned my addr 172.21.15.153:0/3792071940 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf8ed9d640 1 -- 172.21.15.153:0/3792071940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf80009450 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf8ed9d640 1 --2- 172.21.15.153:0/3792071940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbf800317b0 tx=0x7fbf80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf80031e90 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbf8003daf0 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf8003baa0 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.791+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf901115e0 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.792+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf90111fe0 con 0x7fbf90101cb0 2024-06-06T03:09:20.338 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.792+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fbf80039070 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.792+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbf80046990 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.793+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9005ead0 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.795+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fbf80041020 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.854+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fbf90063ed0 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.859+0000 7fbf777fe640 1 -- 172.21.15.153:0/3792071940 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fbf80037070 con 0x7fbf90101cb0 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 msgr2=0x7fbf90110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 --2- 172.21.15.153:0/3792071940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fbf800317b0 tx=0x7fbf80008040 comp rx=0 tx=0).stop 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 shutdown_connections 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 --2- 172.21.15.153:0/3792071940 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf90101cb0 0x7fbf90110ea0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 >> 172.21.15.153:0/3792071940 conn(0x7fbf900fd490 msgr2=0x7fbf900fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 shutdown_connections 2024-06-06T03:09:20.339 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:19.860+0000 7fbf951be640 1 -- 172.21.15.153:0/3792071940 wait complete. 2024-06-06T03:09:20.340 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-06T03:09:20.340 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:09:20.340 INFO:teuthology.orchestra.run.smithi153.stdout:Creating mgr... 2024-06-06T03:09:20.340 INFO:teuthology.orchestra.run.smithi153.stdout:Verifying port 9283 ... 2024-06-06T03:09:20.505 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mgr.smithi153.gidtue 2024-06-06T03:09:20.506 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Failed to reset failed state of unit ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mgr.smithi153.gidtue.service: Unit ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mgr.smithi153.gidtue.service not loaded. 2024-06-06T03:09:20.652 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968.target.wants/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mgr.smithi153.gidtue.service → /etc/systemd/system/ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@.service. 2024-06-06T03:09:20.993 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:20 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3792071940' entity='client.admin' 2024-06-06T03:09:21.155 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T03:09:21.155 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-06-06T03:09:21.163 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T03:09:21.163 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:09:21.163 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-06-06T03:09:21.163 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to enable service . firewalld.service is not available 2024-06-06T03:09:21.172 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T03:09:21.172 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr to start... 2024-06-06T03:09:21.180 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr... 2024-06-06T03:09:22.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:21 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2779213751' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T03:09:22.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:22.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:09:22.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "0b80117e-23b2-11ef-bc9b-c7b262605968", 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:22.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T03:09:22.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T03:09:22.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T03:09:22.247 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.248 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:22.249 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T03:09:14.685937+0000", 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.595+0000 7fb30a093640 1 Processor -- start 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.595+0000 7fb30a093640 1 -- start start 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb30a093640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb30a093640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb304075f70 con 0x7fb304075b60 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:22.250 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33464/0 (socket says 172.21.15.153:33464) 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb3037fe640 1 -- 172.21.15.153:0/690688146 learned_addr learned my addr 172.21.15.153:0/690688146 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.596+0000 7fb3037fe640 1 -- 172.21.15.153:0/690688146 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb304104ec0 con 0x7fb304075b60 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.597+0000 7fb3037fe640 1 --2- 172.21.15.153:0/690688146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009840 tx=0x7fb2f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64cd22af1c0d433a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.597+0000 7fb3027fc640 1 -- 172.21.15.153:0/690688146 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f8032440 con 0x7fb304075b60 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.597+0000 7fb3027fc640 1 -- 172.21.15.153:0/690688146 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2f8032a30 con 0x7fb304075b60 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.597+0000 7fb3027fc640 1 -- 172.21.15.153:0/690688146 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f8031ad0 con 0x7fb304075b60 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 -- 172.21.15.153:0/690688146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 msgr2=0x7fb304104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 --2- 172.21.15.153:0/690688146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8009840 tx=0x7fb2f80091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 -- 172.21.15.153:0/690688146 shutdown_connections 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 --2- 172.21.15.153:0/690688146 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304104980 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 -- 172.21.15.153:0/690688146 >> 172.21.15.153:0/690688146 conn(0x7fb3040fc430 msgr2=0x7fb3040fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 -- 172.21.15.153:0/690688146 shutdown_connections 2024-06-06T03:09:22.251 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 -- 172.21.15.153:0/690688146 wait complete. 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.598+0000 7fb30a093640 1 Processor -- start 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb30a093640 1 -- start start 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb30a093640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb30a093640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb304111040 con 0x7fb304075b60 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb3037fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33466/0 (socket says 172.21.15.153:33466) 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb3037fe640 1 -- 172.21.15.153:0/2779213751 learned_addr learned my addr 172.21.15.153:0/2779213751 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb3037fe640 1 -- 172.21.15.153:0/2779213751 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8009450 con 0x7fb304075b60 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.599+0000 7fb3037fe640 1 --2- 172.21.15.153:0/2779213751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8006f00 tx=0x7fb2f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f803d450 con 0x7fb304075b60 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2f803da40 con 0x7fb304075b60 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb304111240 con 0x7fb304075b60 2024-06-06T03:09:22.252 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2f803bb10 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3041116b0 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fb2f8039070 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.600+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb2f8046e20 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.601+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30404b480 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.603+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fb3041116b0 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.666+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb304111dd0 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.666+0000 7fb2ebfff640 1 -- 172.21.15.153:0/2779213751 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fb2f8041020 con 0x7fb304075b60 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 msgr2=0x7fb304110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 --2- 172.21.15.153:0/2779213751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb2f8006f00 tx=0x7fb2f8008040 comp rx=0 tx=0).stop 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 shutdown_connections 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 --2- 172.21.15.153:0/2779213751 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb304075b60 0x7fb304110b00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 >> 172.21.15.153:0/2779213751 conn(0x7fb3040fc430 msgr2=0x7fb3040fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 shutdown_connections 2024-06-06T03:09:22.253 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:21.667+0000 7fb30a093640 1 -- 172.21.15.153:0/2779213751 wait complete. 2024-06-06T03:09:22.254 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (1/15)... 2024-06-06T03:09:24.897 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:24 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4258729827' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T03:09:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:09:25.283 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "0b80117e-23b2-11ef-bc9b-c7b262605968", 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-06-06T03:09:25.284 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T03:09:25.285 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T03:09:25.293 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.294 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T03:09:14.685937+0000", 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:25.295 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd8902640 1 Processor -- start 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd8902640 1 -- start start 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd8902640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd8902640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd4102700 con 0x7fdfd4101db0 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33468/0 (socket says 172.21.15.153:33468) 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.624+0000 7fdfd259c640 1 -- 172.21.15.153:0/811129387 learned_addr learned my addr 172.21.15.153:0/811129387 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.625+0000 7fdfd259c640 1 -- 172.21.15.153:0/811129387 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4102840 con 0x7fdfd4101db0 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.625+0000 7fdfd259c640 1 --2- 172.21.15.153:0/811129387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009840 tx=0x7fdfc40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92f35fb30481858 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.625+0000 7fdfd1d9b640 1 -- 172.21.15.153:0/811129387 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc4032440 con 0x7fdfd4101db0 2024-06-06T03:09:25.296 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.625+0000 7fdfd1d9b640 1 -- 172.21.15.153:0/811129387 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdfc4032a30 con 0x7fdfd4101db0 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.625+0000 7fdfd1d9b640 1 -- 172.21.15.153:0/811129387 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc4031ad0 con 0x7fdfd4101db0 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 -- 172.21.15.153:0/811129387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 msgr2=0x7fdfd41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 --2- 172.21.15.153:0/811129387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009840 tx=0x7fdfc40091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 -- 172.21.15.153:0/811129387 shutdown_connections 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 --2- 172.21.15.153:0/811129387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd41021c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 -- 172.21.15.153:0/811129387 >> 172.21.15.153:0/811129387 conn(0x7fdfd40fd5b0 msgr2=0x7fdfd40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 -- 172.21.15.153:0/811129387 shutdown_connections 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.626+0000 7fdfd8902640 1 -- 172.21.15.153:0/811129387 wait complete. 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd8902640 1 Processor -- start 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd8902640 1 -- start start 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd8902640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:25.297 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd8902640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfd4110fd0 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33474/0 (socket says 172.21.15.153:33474) 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd259c640 1 -- 172.21.15.153:0/4258729827 learned_addr learned my addr 172.21.15.153:0/4258729827 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd259c640 1 -- 172.21.15.153:0/4258729827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4009450 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.627+0000 7fdfd259c640 1 --2- 172.21.15.153:0/4258729827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009420 tx=0x7fdfc4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc403d5b0 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdfc403dba0 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfd41111d0 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfc403bb40 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfd4111640 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fdfc4039070 con 0x7fdfd4101db0 2024-06-06T03:09:25.298 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.628+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fdfc4046ca0 con 0x7fdfd4101db0 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.629+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfd404b480 con 0x7fdfd4101db0 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.631+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fdfd4111640 con 0x7fdfd4101db0 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.693+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdfd4111d60 con 0x7fdfd4101db0 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.693+0000 7fdfcaffd640 1 -- 172.21.15.153:0/4258729827 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fdfc4041020 con 0x7fdfd4101db0 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 msgr2=0x7fdfd4110a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 --2- 172.21.15.153:0/4258729827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4009420 tx=0x7fdfc4008040 comp rx=0 tx=0).stop 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 shutdown_connections 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 --2- 172.21.15.153:0/4258729827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdfd4101db0 0x7fdfd4110a90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 >> 172.21.15.153:0/4258729827 conn(0x7fdfd40fd5b0 msgr2=0x7fdfd40fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 shutdown_connections 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:24.694+0000 7fdfd8902640 1 -- 172.21.15.153:0/4258729827 wait complete. 2024-06-06T03:09:25.299 INFO:teuthology.orchestra.run.smithi153.stdout:mgr not available, waiting (2/15)... 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: Activating manager daemon smithi153.gidtue 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: mgrmap e2: smithi153.gidtue(active, starting, since 0.00301427s) 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T03:09:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr metadata", "who": "smithi153.gidtue", "id": "smithi153.gidtue"}]: dispatch 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: Manager daemon smithi153.gidtue is now available 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/mirror_snapshot_schedule"}]: dispatch 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/trash_purge_schedule"}]: dispatch 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:26.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:25 smithi153 ceph-mon[29300]: from='mgr.14100 172.21.15.153:0/2139267429' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:27.653 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:27 smithi153 ceph-mon[29300]: mgrmap e3: smithi153.gidtue(active, since 1.00626s) 2024-06-06T03:09:28.473 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:28.473 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsid": "0b80117e-23b2-11ef-bc9b-c7b262605968", 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "health": { 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 0 2024-06-06T03:09:28.474 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "smithi153" 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-06T03:09:28.475 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-06T03:09:28.484 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-06T03:09:28.485 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "restful" 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ], 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "modified": "2024-06-06T03:09:14.685937+0000", 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-06T03:09:28.486 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout }, 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.651+0000 7f52d055f640 1 Processor -- start 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.651+0000 7f52d055f640 1 -- start start 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.651+0000 7f52d055f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.651+0000 7f52d055f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c80708a0 con 0x7f52c8103360 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.652+0000 7f52ce2fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.652+0000 7f52ce2fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33556/0 (socket says 172.21.15.153:33556) 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.652+0000 7f52ce2fb640 1 -- 172.21.15.153:0/2150792608 learned_addr learned my addr 172.21.15.153:0/2150792608 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.652+0000 7f52ce2fb640 1 -- 172.21.15.153:0/2150792608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c8103cb0 con 0x7f52c8103360 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52ce2fb640 1 --2- 172.21.15.153:0/2150792608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009840 tx=0x7f52c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f795de7f42315f74 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52cd2f9640 1 -- 172.21.15.153:0/2150792608 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c4032440 con 0x7f52c8103360 2024-06-06T03:09:28.487 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52cd2f9640 1 -- 172.21.15.153:0/2150792608 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f52c4032a30 con 0x7f52c8103360 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52cd2f9640 1 -- 172.21.15.153:0/2150792608 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c4031ad0 con 0x7f52c8103360 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52d055f640 1 -- 172.21.15.153:0/2150792608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 msgr2=0x7f52c8103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.653+0000 7f52d055f640 1 --2- 172.21.15.153:0/2150792608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f52c4009840 tx=0x7f52c40091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- 172.21.15.153:0/2150792608 shutdown_connections 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 --2- 172.21.15.153:0/2150792608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8103770 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- 172.21.15.153:0/2150792608 >> 172.21.15.153:0/2150792608 conn(0x7f52c80fbc20 msgr2=0x7f52c80fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- 172.21.15.153:0/2150792608 shutdown_connections 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- 172.21.15.153:0/2150792608 wait complete. 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 Processor -- start 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- start start 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52d055f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52c8110fe0 con 0x7f52c8103360 2024-06-06T03:09:28.488 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.654+0000 7f52ce2fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52ce2fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33562/0 (socket says 172.21.15.153:33562) 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52ce2fb640 1 -- 172.21.15.153:0/2643288779 learned_addr learned my addr 172.21.15.153:0/2643288779 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52ce2fb640 1 -- 172.21.15.153:0/2643288779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52c4009450 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52ce2fb640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52c40317b0 tx=0x7f52c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c4031e90 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f52c403daf0 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c81111e0 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52c403baa0 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.655+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c8111650 con 0x7f52c8103360 2024-06-06T03:09:28.489 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.656+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f52c4039070 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.656+0000 7f52beffd640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 0x7f52b4036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.656+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f52c4067430 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.656+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52c805ead0 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.656+0000 7f52cdafa640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 0x7f52b4036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.657+0000 7f52cdafa640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 0x7f52b4036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006fd0 tx=0x7f52b8006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.661+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f52c4041080 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.888+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f52c8111e70 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.888+0000 7f52beffd640 1 -- 172.21.15.153:0/2643288779 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f52c4066090 con 0x7f52c8103360 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 msgr2=0x7f52b4036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 0x7f52b4036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f52b8006fd0 tx=0x7f52b8006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 msgr2=0x7f52c8110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f52c40317b0 tx=0x7f52c4008040 comp rx=0 tx=0).stop 2024-06-06T03:09:28.490 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 shutdown_connections 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f52b40347c0 0x7f52b4036c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 --2- 172.21.15.153:0/2643288779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f52c8103360 0x7f52c8110aa0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 >> 172.21.15.153:0/2643288779 conn(0x7f52c80fbc20 msgr2=0x7f52c80fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 shutdown_connections 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:27.890+0000 7f52d055f640 1 -- 172.21.15.153:0/2643288779 wait complete. 2024-06-06T03:09:28.491 INFO:teuthology.orchestra.run.smithi153.stdout:mgr is available 2024-06-06T03:09:28.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:28 smithi153 ceph-mon[29300]: mgrmap e4: smithi153.gidtue(active, since 2s) 2024-06-06T03:09:28.695 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:28 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2643288779' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [global] 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout fsid = 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-06T03:09:29.534 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout [osd] 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.854+0000 7f240113e640 1 Processor -- start 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.854+0000 7f240113e640 1 -- start start 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f240113e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f240113e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc078120 con 0x7f23fc077d10 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f23fad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f23fad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33564/0 (socket says 172.21.15.153:33564) 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f23fad9d640 1 -- 172.21.15.153:0/4099415823 learned_addr learned my addr 172.21.15.153:0/4099415823 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.855+0000 7f23fad9d640 1 -- 172.21.15.153:0/4099415823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23fc076780 con 0x7f23fc077d10 2024-06-06T03:09:29.535 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f23fad9d640 1 --2- 172.21.15.153:0/4099415823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009840 tx=0x7f23f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d217ed54f96ac38 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f23f9d9b640 1 -- 172.21.15.153:0/4099415823 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f4032440 con 0x7f23fc077d10 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f23f9d9b640 1 -- 172.21.15.153:0/4099415823 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f23f4032a30 con 0x7f23fc077d10 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f23f9d9b640 1 -- 172.21.15.153:0/4099415823 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f4031ad0 con 0x7f23fc077d10 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f240113e640 1 -- 172.21.15.153:0/4099415823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 msgr2=0x7f23fc076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f240113e640 1 --2- 172.21.15.153:0/4099415823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f23f4009840 tx=0x7f23f40091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f240113e640 1 -- 172.21.15.153:0/4099415823 shutdown_connections 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f240113e640 1 --2- 172.21.15.153:0/4099415823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc076240 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.856+0000 7f240113e640 1 -- 172.21.15.153:0/4099415823 >> 172.21.15.153:0/4099415823 conn(0x7f23fc0fd4f0 msgr2=0x7f23fc0ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 -- 172.21.15.153:0/4099415823 shutdown_connections 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 -- 172.21.15.153:0/4099415823 wait complete. 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 Processor -- start 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 -- start start 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f240113e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23fc113570 con 0x7f23fc077d10 2024-06-06T03:09:29.536 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f23fad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f23fad9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33572/0 (socket says 172.21.15.153:33572) 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.857+0000 7f23fad9d640 1 -- 172.21.15.153:0/1950877917 learned_addr learned my addr 172.21.15.153:0/1950877917 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f23fad9d640 1 -- 172.21.15.153:0/1950877917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f4009450 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f23fad9d640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23f4006f00 tx=0x7f23f4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f403d450 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f23f403da40 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23fc113770 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23f403bb10 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.858+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23fc113be0 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.859+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f23f4039070 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.859+0000 7f23f37fe640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 0x7f23e8036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.859+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f23f4067370 con 0x7f23fc077d10 2024-06-06T03:09:29.537 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.859+0000 7f23fa59c640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 0x7f23e8036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.859+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23fc05ead0 con 0x7f23fc077d10 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.860+0000 7f23fa59c640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 0x7f23e8036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006fd0 tx=0x7f23e4006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:28.864+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f23f4037020 con 0x7f23fc077d10 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.028+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f23fc04b480 con 0x7f23fc077d10 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.029+0000 7f23f37fe640 1 -- 172.21.15.153:0/1950877917 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f23f4041070 con 0x7f23fc077d10 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.031+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 msgr2=0x7f23e8036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.031+0000 7f240113e640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 0x7f23e8036c70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f23e4006fd0 tx=0x7f23e4006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 msgr2=0x7f23fc113030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f23f4006f00 tx=0x7f23f4008040 comp rx=0 tx=0).stop 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 shutdown_connections 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f23e80347c0 0x7f23e8036c70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 --2- 172.21.15.153:0/1950877917 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23fc077d10 0x7f23fc113030 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 >> 172.21.15.153:0/1950877917 conn(0x7f23fc0fd4f0 msgr2=0x7f23fc0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:29.538 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 shutdown_connections 2024-06-06T03:09:29.539 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.032+0000 7f240113e640 1 -- 172.21.15.153:0/1950877917 wait complete. 2024-06-06T03:09:29.539 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling cephadm module... 2024-06-06T03:09:29.659 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:29 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1950877917' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-06T03:09:30.866 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:30 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/106783203' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-06T03:09:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:31 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/106783203' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-06T03:09:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:31 smithi153 ceph-mon[29300]: mgrmap e5: smithi153.gidtue(active, since 5s) 2024-06-06T03:09:35.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: Active manager daemon smithi153.gidtue restarted 2024-06-06T03:09:35.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: Activating manager daemon smithi153.gidtue 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: osdmap e2: 0 total, 0 up, 0 in 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: mgrmap e6: smithi153.gidtue(active, starting, since 0.00315742s) 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr metadata", "who": "smithi153.gidtue", "id": "smithi153.gidtue"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: Manager daemon smithi153.gidtue is now available 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:35.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:35.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/mirror_snapshot_schedule"}]: dispatch 2024-06-06T03:09:35.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:35 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/trash_purge_schedule"}]: dispatch 2024-06-06T03:09:36.205 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.912+0000 7f570e30c640 1 Processor -- start 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.912+0000 7f570e30c640 1 -- start start 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f570e30c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f570e30c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57081015a0 con 0x7f5708100c50 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f5707fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f5707fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33588/0 (socket says 172.21.15.153:33588) 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f5707fff640 1 -- 172.21.15.153:0/796502452 learned_addr learned my addr 172.21.15.153:0/796502452 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.913+0000 7f5707fff640 1 -- 172.21.15.153:0/796502452 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57081016e0 con 0x7f5708100c50 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f5707fff640 1 --2- 172.21.15.153:0/796502452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009840 tx=0x7f56f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff7a66127ba0d10a server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f5706ffd640 1 -- 172.21.15.153:0/796502452 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f8032440 con 0x7f5708100c50 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f5706ffd640 1 -- 172.21.15.153:0/796502452 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f56f8032a30 con 0x7f5708100c50 2024-06-06T03:09:36.206 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f5706ffd640 1 -- 172.21.15.153:0/796502452 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f8031ad0 con 0x7f5708100c50 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f570e30c640 1 -- 172.21.15.153:0/796502452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 msgr2=0x7f5708101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f570e30c640 1 --2- 172.21.15.153:0/796502452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009840 tx=0x7f56f80091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f570e30c640 1 -- 172.21.15.153:0/796502452 shutdown_connections 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f570e30c640 1 --2- 172.21.15.153:0/796502452 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708101060 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.914+0000 7f570e30c640 1 -- 172.21.15.153:0/796502452 >> 172.21.15.153:0/796502452 conn(0x7f57080fc430 msgr2=0x7f57080fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 -- 172.21.15.153:0/796502452 shutdown_connections 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 -- 172.21.15.153:0/796502452 wait complete. 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 Processor -- start 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 -- start start 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.915+0000 7f570e30c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708113810 con 0x7f5708100c50 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5707fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:36.207 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5707fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33596/0 (socket says 172.21.15.153:33596) 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5707fff640 1 -- 172.21.15.153:0/106783203 learned_addr learned my addr 172.21.15.153:0/106783203 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5707fff640 1 -- 172.21.15.153:0/106783203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f8009450 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5707fff640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009420 tx=0x7f56f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f803d5b0 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f56f803dba0 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5708113950 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56f803bb40 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.916+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5708113dc0 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.917+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f56f8039070 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.917+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570804b480 con 0x7f5708100c50 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.917+0000 7f5704ff9640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 0x7f56f0036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:36.208 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.917+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f56f8067610 con 0x7f5708100c50 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.917+0000 7f57077fe640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 0x7f56f0036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.918+0000 7f57077fe640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 0x7f56f0036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56fc006fd0 tx=0x7f56fc006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:29.922+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f56f8039320 con 0x7f5708100c50 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.134+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f5708114000 con 0x7f5708100c50 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.601+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f56f8046780 con 0x7f5708100c50 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.602+0000 7f5704ff9640 1 -- 172.21.15.153:0/106783203 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f56f8069710 con 0x7f5708100c50 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 msgr2=0x7f56f0036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 0x7f56f0036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f56fc006fd0 tx=0x7f56fc006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 msgr2=0x7f5708075b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f56f8009420 tx=0x7f56f8008040 comp rx=0 tx=0).stop 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 shutdown_connections 2024-06-06T03:09:36.209 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:6800/3563458677,v1:172.21.15.153:6801/3563458677] conn(0x7f56f00347c0 0x7f56f0036c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:36.210 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 --2- 172.21.15.153:0/106783203 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5708100c50 0x7f5708075b10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:36.210 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 >> 172.21.15.153:0/106783203 conn(0x7f57080fc430 msgr2=0x7f57080fcfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:36.210 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 shutdown_connections 2024-06-06T03:09:36.210 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:30.606+0000 7f570e30c640 1 -- 172.21.15.153:0/106783203 wait complete. 2024-06-06T03:09:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:36 smithi153 ceph-mon[29300]: Found migration_current of "None". Setting to last migration. 2024-06-06T03:09:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:36 smithi153 ceph-mon[29300]: mgrmap e7: smithi153.gidtue(active, since 1.00465s) 2024-06-06T03:09:38.194 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: [06/Jun/2024:03:09:36] ENGINE Bus STARTING 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: [06/Jun/2024:03:09:36] ENGINE Serving on https://172.21.15.153:7150 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: [06/Jun/2024:03:09:36] ENGINE Bus STARTED 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: [06/Jun/2024:03:09:36] ENGINE Error in HTTPServer.serve 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Traceback (most recent call last): 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._run(expiration_interval) 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T03:09:38.195 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s = self.context.wrap_socket( 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: return self.sslsocket_class._create( 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self.do_handshake() 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._sslobj.do_handshake() 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:38.196 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:37 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:39.068 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:38 smithi153 ceph-mon[29300]: mgrmap e8: smithi153.gidtue(active, since 2s) 2024-06-06T03:09:40.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:39 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1418960771' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T03:09:42.075 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 8, 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.gidtue", 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e8d55640 1 Processor -- start 2024-06-06T03:09:42.081 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e8d55640 1 -- start start 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e8d55640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e8d55640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4102580 con 0x7f03e4101c30 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39380/0 (socket says 172.21.15.153:39380) 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.155+0000 7f03e259c640 1 -- 172.21.15.153:0/1925108324 learned_addr learned my addr 172.21.15.153:0/1925108324 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.156+0000 7f03e259c640 1 -- 172.21.15.153:0/1925108324 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03e41026c0 con 0x7f03e4101c30 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.156+0000 7f03e259c640 1 --2- 172.21.15.153:0/1925108324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009840 tx=0x7f03d40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c8943297d210ca server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e1d9b640 1 -- 172.21.15.153:0/1925108324 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d4032440 con 0x7f03e4101c30 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e1d9b640 1 -- 172.21.15.153:0/1925108324 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f03d4032a30 con 0x7f03e4101c30 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e1d9b640 1 -- 172.21.15.153:0/1925108324 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d4031ad0 con 0x7f03e4101c30 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 -- 172.21.15.153:0/1925108324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 msgr2=0x7f03e4102040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:42.082 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1925108324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f03d4009840 tx=0x7f03d40091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 -- 172.21.15.153:0/1925108324 shutdown_connections 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1925108324 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4102040 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 -- 172.21.15.153:0/1925108324 >> 172.21.15.153:0/1925108324 conn(0x7f03e40fd430 msgr2=0x7f03e40ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 -- 172.21.15.153:0/1925108324 shutdown_connections 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.157+0000 7f03e8d55640 1 -- 172.21.15.153:0/1925108324 wait complete. 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e8d55640 1 Processor -- start 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e8d55640 1 -- start start 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e8d55640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e8d55640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03e4110fc0 con 0x7f03e4101c30 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39388/0 (socket says 172.21.15.153:39388) 2024-06-06T03:09:42.083 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e259c640 1 -- 172.21.15.153:0/1418960771 learned_addr learned my addr 172.21.15.153:0/1418960771 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e259c640 1 -- 172.21.15.153:0/1418960771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d4009450 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.158+0000 7f03e259c640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03d40317b0 tx=0x7f03d4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.159+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d4031e90 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.159+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f03d403daf0 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.159+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03e41111c0 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.159+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f03d403baa0 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.159+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03e4111630 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f03d4039070 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03e404b480 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03daffd640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 0x7f03d0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f03d40676a0 con 0x7f03e4101c30 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03dbfff640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 0x7f03d0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:42.084 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.160+0000 7f03dbfff640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 0x7f03d0036d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f03cc006fd0 tx=0x7f03cc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.165+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f03d403ca00 con 0x7f03e4101c30 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.373+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f03e4103a90 con 0x7f03e4101c30 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.373+0000 7f03daffd640 1 -- 172.21.15.153:0/1418960771 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v8) v1 ==== 56+0+103 (secure 0 0 0) 0x7f03d4041070 con 0x7f03e4101c30 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 msgr2=0x7f03d0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 0x7f03d0036d10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f03cc006fd0 tx=0x7f03cc006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 msgr2=0x7f03e4110a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f03d40317b0 tx=0x7f03d4008040 comp rx=0 tx=0).stop 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 shutdown_connections 2024-06-06T03:09:42.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f03d0034860 0x7f03d0036d10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 --2- 172.21.15.153:0/1418960771 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f03e4101c30 0x7f03e4110a80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 >> 172.21.15.153:0/1418960771 conn(0x7f03e40fd430 msgr2=0x7f03e40fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 shutdown_connections 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:39.375+0000 7f03e8d55640 1 -- 172.21.15.153:0/1418960771 wait complete. 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-06-06T03:09:42.086 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 8... 2024-06-06T03:09:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:42 smithi153 ceph-mon[29300]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-06T03:09:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:42 smithi153 ceph-mon[29300]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.454+0000 7fdd4a653640 1 Processor -- start 2024-06-06T03:09:43.085 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd4a653640 1 -- start start 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd4a653640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd4a653640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44075f70 con 0x7fdd44075b60 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55642/0 (socket says 172.21.15.153:55642) 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.455+0000 7fdd43fff640 1 -- 172.21.15.153:0/3642710030 learned_addr learned my addr 172.21.15.153:0/3642710030 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.456+0000 7fdd43fff640 1 -- 172.21.15.153:0/3642710030 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd44104ec0 con 0x7fdd44075b60 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.456+0000 7fdd43fff640 1 --2- 172.21.15.153:0/3642710030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009840 tx=0x7fdd340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07075be6cc5fb46 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.456+0000 7fdd42ffd640 1 -- 172.21.15.153:0/3642710030 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34032440 con 0x7fdd44075b60 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.456+0000 7fdd42ffd640 1 -- 172.21.15.153:0/3642710030 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdd34032a30 con 0x7fdd44075b60 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd42ffd640 1 -- 172.21.15.153:0/3642710030 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd34031ad0 con 0x7fdd44075b60 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 -- 172.21.15.153:0/3642710030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 msgr2=0x7fdd44104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 --2- 172.21.15.153:0/3642710030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdd34009840 tx=0x7fdd340091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:43.086 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 -- 172.21.15.153:0/3642710030 shutdown_connections 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 --2- 172.21.15.153:0/3642710030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44104980 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 -- 172.21.15.153:0/3642710030 >> 172.21.15.153:0/3642710030 conn(0x7fdd440fc430 msgr2=0x7fdd440fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 -- 172.21.15.153:0/3642710030 shutdown_connections 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.457+0000 7fdd4a653640 1 -- 172.21.15.153:0/3642710030 wait complete. 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd4a653640 1 Processor -- start 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd4a653640 1 -- start start 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd4a653640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd4a653640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44111060 con 0x7fdd44075b60 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55646/0 (socket says 172.21.15.153:55646) 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.458+0000 7fdd43fff640 1 -- 172.21.15.153:0/1319442281 learned_addr learned my addr 172.21.15.153:0/1319442281 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd43fff640 1 -- 172.21.15.153:0/1319442281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34009450 con 0x7fdd44075b60 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd43fff640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd34006f00 tx=0x7fdd34008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:43.087 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3403d450 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fdd3403da40 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3403bb10 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd44111260 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.459+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd441116d0 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fdd34039070 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd40ff9640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 0x7fdd2c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdd441116d0 con 0x7fdd2c034860 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdd34067590 con 0x7fdd44075b60 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd437fe640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 0x7fdd2c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.460+0000 7fdd437fe640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 0x7fdd2c036d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd38006fd0 tx=0x7fdd38006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.461+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fdd441116d0 con 0x7fdd2c034860 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fdd4404b480 con 0x7fdd2c034860 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd40ff9640 1 -- 172.21.15.153:0/1319442281 <== mgr.14118 v2:172.21.15.153:6800/1921588694 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fdd4404b480 con 0x7fdd2c034860 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 msgr2=0x7fdd2c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 0x7fdd2c036d10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fdd38006fd0 tx=0x7fdd38006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 msgr2=0x7fdd44110b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd34006f00 tx=0x7fdd34008040 comp rx=0 tx=0).stop 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 shutdown_connections 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fdd2c034860 0x7fdd2c036d10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 --2- 172.21.15.153:0/1319442281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd44075b60 0x7fdd44110b20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.466+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 >> 172.21.15.153:0/1319442281 conn(0x7fdd440fc430 msgr2=0x7fdd440fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.467+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 shutdown_connections 2024-06-06T03:09:43.089 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:42.467+0000 7fdd4a653640 1 -- 172.21.15.153:0/1319442281 wait complete. 2024-06-06T03:09:43.090 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 8 is available 2024-06-06T03:09:43.090 INFO:teuthology.orchestra.run.smithi153.stdout:Setting orchestrator backend to cephadm... 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.481+0000 7f4930b08640 1 Processor -- start 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f4930b08640 1 -- start start 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f4930b08640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f4930b08640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c1026e0 con 0x7f492c101d90 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f492a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f492a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55658/0 (socket says 172.21.15.153:55658) 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f492a59c640 1 -- 172.21.15.153:0/1538716969 learned_addr learned my addr 172.21.15.153:0/1538716969 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.482+0000 7f492a59c640 1 -- 172.21.15.153:0/1538716969 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f492c102820 con 0x7f492c101d90 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.483+0000 7f492a59c640 1 --2- 172.21.15.153:0/1538716969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f491c009840 tx=0x7f491c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f2ca1f299dec6b server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:44.234 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.483+0000 7f492959a640 1 -- 172.21.15.153:0/1538716969 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f491c032440 con 0x7f492c101d90 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.483+0000 7f492959a640 1 -- 172.21.15.153:0/1538716969 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f491c032a30 con 0x7f492c101d90 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.483+0000 7f492959a640 1 -- 172.21.15.153:0/1538716969 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f491c031ad0 con 0x7f492c101d90 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- 172.21.15.153:0/1538716969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 msgr2=0x7f492c1021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 --2- 172.21.15.153:0/1538716969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f491c009840 tx=0x7f491c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- 172.21.15.153:0/1538716969 shutdown_connections 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 --2- 172.21.15.153:0/1538716969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c1021a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- 172.21.15.153:0/1538716969 >> 172.21.15.153:0/1538716969 conn(0x7f492c0fd5b0 msgr2=0x7f492c0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- 172.21.15.153:0/1538716969 shutdown_connections 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- 172.21.15.153:0/1538716969 wait complete. 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 Processor -- start 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- start start 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:44.235 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.484+0000 7f4930b08640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c078100 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f492a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f492a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55666/0 (socket says 172.21.15.153:55666) 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f492a59c640 1 -- 172.21.15.153:0/3376180884 learned_addr learned my addr 172.21.15.153:0/3376180884 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f492a59c640 1 -- 172.21.15.153:0/3376180884 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c009450 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f492a59c640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f491c0317b0 tx=0x7f491c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f491c031e90 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f491c03daf0 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f491c03baa0 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c0761e0 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.485+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c076650 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.486+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f491c039070 con 0x7f492c101d90 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.486+0000 7f491affd640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 0x7f4914036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:44.236 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.486+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f491c067860 con 0x7f492c101d90 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.486+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f492c05ead0 con 0x7f492c101d90 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.486+0000 7f4929d9b640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 0x7f4914036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.487+0000 7f4929d9b640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 0x7f4914036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4920006fd0 tx=0x7f4920006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.491+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f491c041080 con 0x7f492c101d90 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.665+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f492c103c10 con 0x7f4914034860 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.673+0000 7f491affd640 1 -- 172.21.15.153:0/3376180884 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f492c103c10 con 0x7f4914034860 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 msgr2=0x7f4914036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 0x7f4914036d10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4920006fd0 tx=0x7f4920006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 msgr2=0x7f492c077bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f491c0317b0 tx=0x7f491c008040 comp rx=0 tx=0).stop 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 shutdown_connections 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4914034860 0x7f4914036d10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 --2- 172.21.15.153:0/3376180884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f492c101d90 0x7f492c077bc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:44.237 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 >> 172.21.15.153:0/3376180884 conn(0x7f492c0fd5b0 msgr2=0x7f492c0fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:44.238 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 shutdown_connections 2024-06-06T03:09:44.238 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:43.675+0000 7f4930b08640 1 -- 172.21.15.153:0/3376180884 wait complete. 2024-06-06T03:09:44.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:44 smithi153 ceph-mon[29300]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:44.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:44 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:44.805 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:44 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:45.373 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-06T03:09:45.373 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.628+0000 7f0451f50640 1 Processor -- start 2024-06-06T03:09:45.373 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.628+0000 7f0451f50640 1 -- start start 2024-06-06T03:09:45.373 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.628+0000 7f0451f50640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.629+0000 7f0451f50640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c078030 con 0x7f044c077c20 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.629+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.629+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55680/0 (socket says 172.21.15.153:55680) 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.629+0000 7f044b7fe640 1 -- 172.21.15.153:0/1478494865 learned_addr learned my addr 172.21.15.153:0/1478494865 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.629+0000 7f044b7fe640 1 -- 172.21.15.153:0/1478494865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c076780 con 0x7f044c077c20 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f044b7fe640 1 --2- 172.21.15.153:0/1478494865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f043c009840 tx=0x7f043c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf3ac03a576d804 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f044a7fc640 1 -- 172.21.15.153:0/1478494865 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c032440 con 0x7f044c077c20 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f044a7fc640 1 -- 172.21.15.153:0/1478494865 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f043c032a30 con 0x7f044c077c20 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f044a7fc640 1 -- 172.21.15.153:0/1478494865 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c031ad0 con 0x7f044c077c20 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 -- 172.21.15.153:0/1478494865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 msgr2=0x7f044c076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 --2- 172.21.15.153:0/1478494865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f043c009840 tx=0x7f043c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:45.374 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 -- 172.21.15.153:0/1478494865 shutdown_connections 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 --2- 172.21.15.153:0/1478494865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c076240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 -- 172.21.15.153:0/1478494865 >> 172.21.15.153:0/1478494865 conn(0x7f044c0fd590 msgr2=0x7f044c0ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.630+0000 7f0451f50640 1 -- 172.21.15.153:0/1478494865 shutdown_connections 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f0451f50640 1 -- 172.21.15.153:0/1478494865 wait complete. 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f0451f50640 1 Processor -- start 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f0451f50640 1 -- start start 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f0451f50640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f0451f50640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044c113580 con 0x7f044c077c20 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f044b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55684/0 (socket says 172.21.15.153:55684) 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.631+0000 7f044b7fe640 1 -- 172.21.15.153:0/1197275085 learned_addr learned my addr 172.21.15.153:0/1197275085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f044b7fe640 1 -- 172.21.15.153:0/1197275085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c009450 con 0x7f044c077c20 2024-06-06T03:09:45.375 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f044b7fe640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f043c031b90 tx=0x7f043c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c0328b0 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f043c03d8e0 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c113780 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f043c03ba20 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.632+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044c113bf0 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f043c039070 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044c04b480 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f0443fff640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 0x7f0438036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f043c067790 con 0x7f044c077c20 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f044affd640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 0x7f0438036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.633+0000 7f044affd640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 0x7f0438036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0434006fd0 tx=0x7f0434006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:45.376 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.638+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f043c04e350 con 0x7f044c077c20 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.805+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f044c113bf0 con 0x7f0438034860 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.805+0000 7f0443fff640 1 -- 172.21.15.153:0/1197275085 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f044c113bf0 con 0x7f0438034860 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 msgr2=0x7f0438036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 0x7f0438036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0434006fd0 tx=0x7f0434006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 msgr2=0x7f044c113040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f043c031b90 tx=0x7f043c008040 comp rx=0 tx=0).stop 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 shutdown_connections 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f0438034860 0x7f0438036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 --2- 172.21.15.153:0/1197275085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f044c077c20 0x7f044c113040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 >> 172.21.15.153:0/1197275085 conn(0x7f044c0fd590 msgr2=0x7f044c0fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 shutdown_connections 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:44.808+0000 7f0451f50640 1 -- 172.21.15.153:0/1197275085 wait complete. 2024-06-06T03:09:45.377 INFO:teuthology.orchestra.run.smithi153.stdout:Generating ssh key... 2024-06-06T03:09:46.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:45 smithi153 ceph-mon[29300]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:46.579 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.742+0000 7f4230c6d640 1 Processor -- start 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.742+0000 7f4230c6d640 1 -- start start 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f4230c6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f4230c6d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c075f20 con 0x7f422c075b10 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f422a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f422a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55694/0 (socket says 172.21.15.153:55694) 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f422a59c640 1 -- 172.21.15.153:0/4037036274 learned_addr learned my addr 172.21.15.153:0/4037036274 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.743+0000 7f422a59c640 1 -- 172.21.15.153:0/4037036274 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f422c104f20 con 0x7f422c075b10 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f422a59c640 1 --2- 172.21.15.153:0/4037036274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f421c009840 tx=0x7f421c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccba8f22a52f69f6 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f4229d9b640 1 -- 172.21.15.153:0/4037036274 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c032440 con 0x7f422c075b10 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f4229d9b640 1 -- 172.21.15.153:0/4037036274 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f421c032a30 con 0x7f422c075b10 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f4229d9b640 1 -- 172.21.15.153:0/4037036274 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c031ad0 con 0x7f422c075b10 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f4230c6d640 1 -- 172.21.15.153:0/4037036274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 msgr2=0x7f422c1049e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:46.580 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.744+0000 7f4230c6d640 1 --2- 172.21.15.153:0/4037036274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f421c009840 tx=0x7f421c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- 172.21.15.153:0/4037036274 shutdown_connections 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 --2- 172.21.15.153:0/4037036274 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c1049e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- 172.21.15.153:0/4037036274 >> 172.21.15.153:0/4037036274 conn(0x7f422c0fc490 msgr2=0x7f422c0fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- 172.21.15.153:0/4037036274 shutdown_connections 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- 172.21.15.153:0/4037036274 wait complete. 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 Processor -- start 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- start start 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f4230c6d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422c111000 con 0x7f422c075b10 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f422a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f422a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55710/0 (socket says 172.21.15.153:55710) 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.745+0000 7f422a59c640 1 -- 172.21.15.153:0/3138638789 learned_addr learned my addr 172.21.15.153:0/3138638789 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f422a59c640 1 -- 172.21.15.153:0/3138638789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f421c009450 con 0x7f422c075b10 2024-06-06T03:09:46.581 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f422a59c640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f421c031b90 tx=0x7f421c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c0328b0 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f421c03d8e0 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f422c111200 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f421c03ba20 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.746+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f422c111670 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f421c039070 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f422c04b480 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4222ffd640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 0x7f4218036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f421c067780 con 0x7f422c075b10 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4223fff640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 0x7f4218036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.747+0000 7f4223fff640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 0x7f4218036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4214006fd0 tx=0x7f4214006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:46.582 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.751+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f421c04ebb0 con 0x7f422c075b10 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:45.915+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f422c111670 con 0x7f4218034860 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.054+0000 7f4222ffd640 1 -- 172.21.15.153:0/3138638789 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f422c111670 con 0x7f4218034860 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 msgr2=0x7f4218036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 0x7f4218036d10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4214006fd0 tx=0x7f4214006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 msgr2=0x7f422c110ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f421c031b90 tx=0x7f421c008040 comp rx=0 tx=0).stop 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 shutdown_connections 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f4218034860 0x7f4218036d10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 --2- 172.21.15.153:0/3138638789 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f422c075b10 0x7f422c110ac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 >> 172.21.15.153:0/3138638789 conn(0x7f422c0fc490 msgr2=0x7f422c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 shutdown_connections 2024-06-06T03:09:46.583 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.056+0000 7f4230c6d640 1 -- 172.21.15.153:0/3138638789 wait complete. 2024-06-06T03:09:47.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:47 smithi153 ceph-mon[29300]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:47.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:47 smithi153 ceph-mon[29300]: Generating ssh key... 2024-06-06T03:09:47.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:47 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:47.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:47 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4yiWuyIIQ+5WRiIVB23IHvsfHZwbjWTQdQ2/EEhL8tKfekI5YiQJXqjm4QCY96RwuqJXin72ifSnnUXEbyzJYW3VcBTlPnGgEPmltcjC2fRF6w68GJoo3fYgxqYv50qHEaOx1IhLifrL5/IxSGe/1bj4jmKYVjlclg4IvsEhfzzTnZls0j+MIOPZSz7bWAdy3WNdkG5zlCN49m34mHTaZ+Z5QDtqsd/2oZ49AR/QyN7CTXcnBzMo4dr0K15BHVS9c511ffSYFecrLvjjZwhitdahmpaQMB7NsAQbe5GsebN4++o2gL9BLegcIrZfYODXugq0NnR0hWyxGtoKfzifW2BaDZQZmEQ8fihIudZ4uTR7v7LGNJn/yB+/ashDcuK3B2ABxkVJR/qIZScaCLNy9ZIlsJHNFyt3D0fwEcMVdIWZVgXGzmCC8enrL6zIa4B6x69EBTKLRvaLuBtz33E25N95QnttP+N/hCdiZ54EzQZZ83tJTZrm8lpxn97jRCYE= ceph-0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f8558f0b640 1 Processor -- start 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f8558f0b640 1 -- start start 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f8558f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f8558f0b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554078030 con 0x7f8554077c20 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f855259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f855259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55726/0 (socket says 172.21.15.153:55726) 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.935+0000 7f855259c640 1 -- 172.21.15.153:0/958571896 learned_addr learned my addr 172.21.15.153:0/958571896 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.936+0000 7f855259c640 1 -- 172.21.15.153:0/958571896 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554076780 con 0x7f8554077c20 2024-06-06T03:09:47.904 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.936+0000 7f855259c640 1 --2- 172.21.15.153:0/958571896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8544009840 tx=0x7f85440091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9797d82605825406 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.936+0000 7f8551d9b640 1 -- 172.21.15.153:0/958571896 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8544032440 con 0x7f8554077c20 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8551d9b640 1 -- 172.21.15.153:0/958571896 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f8544032a30 con 0x7f8554077c20 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8551d9b640 1 -- 172.21.15.153:0/958571896 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8544031ad0 con 0x7f8554077c20 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 -- 172.21.15.153:0/958571896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 msgr2=0x7f8554076240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 --2- 172.21.15.153:0/958571896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8544009840 tx=0x7f85440091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 -- 172.21.15.153:0/958571896 shutdown_connections 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 --2- 172.21.15.153:0/958571896 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554076240 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 -- 172.21.15.153:0/958571896 >> 172.21.15.153:0/958571896 conn(0x7f85540fd590 msgr2=0x7f85540ff9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 -- 172.21.15.153:0/958571896 shutdown_connections 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.937+0000 7f8558f0b640 1 -- 172.21.15.153:0/958571896 wait complete. 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f8558f0b640 1 Processor -- start 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f8558f0b640 1 -- start start 2024-06-06T03:09:47.905 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f8558f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f8558f0b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554113250 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f855259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f855259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55730/0 (socket says 172.21.15.153:55730) 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.938+0000 7f855259c640 1 -- 172.21.15.153:0/3845181038 learned_addr learned my addr 172.21.15.153:0/3845181038 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f855259c640 1 -- 172.21.15.153:0/3845181038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8544009450 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f855259c640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8544031b90 tx=0x7f8544008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85440328b0 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554113450 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f854403d8e0 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f854403ba20 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.939+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85541138c0 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.940+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8544039070 con 0x7f8554077c20 2024-06-06T03:09:47.906 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.940+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855404b480 con 0x7f8554077c20 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.940+0000 7f854affd640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 0x7f8540036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.940+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f85440675f0 con 0x7f8554077c20 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.940+0000 7f854bfff640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 0x7f8540036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.941+0000 7f854bfff640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 0x7f8540036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f853c006fd0 tx=0x7f853c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:46.945+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f854404ebb0 con 0x7f8554077c20 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.116+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f85541138c0 con 0x7f8540034860 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.116+0000 7f854affd640 1 -- 172.21.15.153:0/3845181038 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f85541138c0 con 0x7f8540034860 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 msgr2=0x7f8540036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 0x7f8540036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f853c006fd0 tx=0x7f853c006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 msgr2=0x7f8554112d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8544031b90 tx=0x7f8544008040 comp rx=0 tx=0).stop 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 shutdown_connections 2024-06-06T03:09:47.907 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8540034860 0x7f8540036d10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 --2- 172.21.15.153:0/3845181038 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8554077c20 0x7f8554112d10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.118+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 >> 172.21.15.153:0/3845181038 conn(0x7f85540fd590 msgr2=0x7f85540fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.119+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 shutdown_connections 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:47.119+0000 7f8558f0b640 1 -- 172.21.15.153:0/3845181038 wait complete. 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:Adding key to root@localhost authorized_keys... 2024-06-06T03:09:47.908 INFO:teuthology.orchestra.run.smithi153.stdout:Adding host smithi153... 2024-06-06T03:09:48.430 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:48 smithi153 ceph-mon[29300]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:49.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:49 smithi153 ceph-mon[29300]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:50.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:50 smithi153 ceph-mon[29300]: Deploying cephadm binary to smithi153 2024-06-06T03:09:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Added host 'smithi153' with addr '172.21.15.153' 2024-06-06T03:09:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.254+0000 7ff9736c3640 1 Processor -- start 2024-06-06T03:09:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.254+0000 7ff9736c3640 1 -- start start 2024-06-06T03:09:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.254+0000 7ff9736c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:51.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.254+0000 7ff9736c3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1015a0 con 0x7ff96c100c50 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.255+0000 7ff97145f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.255+0000 7ff97145f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55738/0 (socket says 172.21.15.153:55738) 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.255+0000 7ff97145f640 1 -- 172.21.15.153:0/523156083 learned_addr learned my addr 172.21.15.153:0/523156083 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.255+0000 7ff97145f640 1 -- 172.21.15.153:0/523156083 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c1016e0 con 0x7ff96c100c50 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff97145f640 1 --2- 172.21.15.153:0/523156083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff96800baf0 tx=0x7ff96800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f980caa506f3a6f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff963fff640 1 -- 172.21.15.153:0/523156083 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff9680327a0 con 0x7ff96c100c50 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff963fff640 1 -- 172.21.15.153:0/523156083 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff968032d90 con 0x7ff96c100c50 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff963fff640 1 -- 172.21.15.153:0/523156083 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff968031e70 con 0x7ff96c100c50 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff9736c3640 1 -- 172.21.15.153:0/523156083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 msgr2=0x7ff96c101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff9736c3640 1 --2- 172.21.15.153:0/523156083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff96800baf0 tx=0x7ff96800b460 comp rx=0 tx=0).stop 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff9736c3640 1 -- 172.21.15.153:0/523156083 shutdown_connections 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff9736c3640 1 --2- 172.21.15.153:0/523156083 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c101060 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:51.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.256+0000 7ff9736c3640 1 -- 172.21.15.153:0/523156083 >> 172.21.15.153:0/523156083 conn(0x7ff96c0fc430 msgr2=0x7ff96c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 -- 172.21.15.153:0/523156083 shutdown_connections 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 -- 172.21.15.153:0/523156083 wait complete. 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 Processor -- start 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 -- start start 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff9736c3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c10eec0 con 0x7ff96c100c50 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff97145f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff97145f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55754/0 (socket says 172.21.15.153:55754) 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.257+0000 7ff97145f640 1 -- 172.21.15.153:0/602103677 learned_addr learned my addr 172.21.15.153:0/602103677 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff97145f640 1 -- 172.21.15.153:0/602103677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96800b700 con 0x7ff96c100c50 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff97145f640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff968006f00 tx=0x7ff96800b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff968032a30 con 0x7ff96c100c50 2024-06-06T03:09:51.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7ff968032b90 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96c10f0c0 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff96803a730 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.258+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96c10f4d0 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.259+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff96803bc20 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.259+0000 7ff961ffb640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 0x7ff958036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.259+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff968069b50 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.259+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff96c04b480 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.259+0000 7ff970c5e640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 0x7ff958036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.260+0000 7ff970c5e640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 0x7ff958036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff95c006fd0 tx=0x7ff95c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.264+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff96803b420 con 0x7ff96c100c50 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:48.430+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi153", "addr": "172.21.15.153", "target": ["mon-mgr", ""]}) v1 -- 0x7ff96c10ffc0 con 0x7ff958034810 2024-06-06T03:09:51.806 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.218+0000 7ff961ffb640 1 -- 172.21.15.153:0/602103677 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7ff96c10ffc0 con 0x7ff958034810 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.221+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 msgr2=0x7ff958036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.221+0000 7ff9736c3640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 0x7ff958036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff95c006fd0 tx=0x7ff95c006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.221+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 msgr2=0x7ff96c10e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.221+0000 7ff9736c3640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff968006f00 tx=0x7ff96800b460 comp rx=0 tx=0).stop 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 shutdown_connections 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff958034810 0x7ff958036cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 --2- 172.21.15.153:0/602103677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff96c100c50 0x7ff96c10e980 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 >> 172.21.15.153:0/602103677 conn(0x7ff96c0fc430 msgr2=0x7ff96c105200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 shutdown_connections 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:51.222+0000 7ff9736c3640 1 -- 172.21.15.153:0/602103677 wait complete. 2024-06-06T03:09:51.807 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mon service with default placement... 2024-06-06T03:09:52.442 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:52 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:52.443 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:52 smithi153 ceph-mon[29300]: Added host smithi153 2024-06-06T03:09:52.443 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:52 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.252+0000 7f1cc861b640 1 Processor -- start 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.252+0000 7f1cc861b640 1 -- start start 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc861b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc861b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc0102600 con 0x7f1cc0101cb0 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc63b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc63b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47688/0 (socket says 172.21.15.153:47688) 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc63b7640 1 -- 172.21.15.153:0/3665828780 learned_addr learned my addr 172.21.15.153:0/3665828780 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.253+0000 7f1cc63b7640 1 -- 172.21.15.153:0/3665828780 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc0102740 con 0x7f1cc0101cb0 2024-06-06T03:09:53.173 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc63b7640 1 --2- 172.21.15.153:0/3665828780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009840 tx=0x7f1cb00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee86d841f3e4d69 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc53b5640 1 -- 172.21.15.153:0/3665828780 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb0032440 con 0x7f1cc0101cb0 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc53b5640 1 -- 172.21.15.153:0/3665828780 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1cb0032a30 con 0x7f1cc0101cb0 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc53b5640 1 -- 172.21.15.153:0/3665828780 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb0031ad0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc861b640 1 -- 172.21.15.153:0/3665828780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 msgr2=0x7f1cc01020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.254+0000 7f1cc861b640 1 --2- 172.21.15.153:0/3665828780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009840 tx=0x7f1cb00091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- 172.21.15.153:0/3665828780 shutdown_connections 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 --2- 172.21.15.153:0/3665828780 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc01020c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- 172.21.15.153:0/3665828780 >> 172.21.15.153:0/3665828780 conn(0x7f1cc00fd490 msgr2=0x7f1cc00ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- 172.21.15.153:0/3665828780 shutdown_connections 2024-06-06T03:09:53.174 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- 172.21.15.153:0/3665828780 wait complete. 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 Processor -- start 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- start start 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc861b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc01113f0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc63b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc63b7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47692/0 (socket says 172.21.15.153:47692) 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.255+0000 7f1cc63b7640 1 -- 172.21.15.153:0/813645048 learned_addr learned my addr 172.21.15.153:0/813645048 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cc63b7640 1 -- 172.21.15.153:0/813645048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0009450 con 0x7f1cc0101cb0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cc63b7640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00317b0 tx=0x7f1cb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb0031e90 con 0x7f1cc0101cb0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1cb003daf0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc01115f0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb003baa0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.256+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc0111ff0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1cb0039070 con 0x7f1cc0101cb0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cb6ffd640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 0x7f1cac036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1cb0067920 con 0x7f1cc0101cb0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc005ead0 con 0x7f1cc0101cb0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cc5bb6640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 0x7f1cac036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:53.176 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.257+0000 7f1cc5bb6640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 0x7f1cac036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc006fd0 tx=0x7f1cbc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.262+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1cb0041080 con 0x7f1cc0101cb0 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.442+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cc0103c20 con 0x7f1cac034860 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.449+0000 7f1cb6ffd640 1 -- 172.21.15.153:0/813645048 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1cc0103c20 con 0x7f1cac034860 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 msgr2=0x7f1cac036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 0x7f1cac036d10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc006fd0 tx=0x7f1cbc006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 msgr2=0x7f1cc0110eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1cb00317b0 tx=0x7f1cb0008040 comp rx=0 tx=0).stop 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 shutdown_connections 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1cac034860 0x7f1cac036d10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 --2- 172.21.15.153:0/813645048 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1cc0101cb0 0x7f1cc0110eb0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 >> 172.21.15.153:0/813645048 conn(0x7f1cc00fd490 msgr2=0x7f1cc00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:53.177 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 shutdown_connections 2024-06-06T03:09:53.178 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:52.452+0000 7f1cc861b640 1 -- 172.21.15.153:0/813645048 wait complete. 2024-06-06T03:09:53.178 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying mgr service with default placement... 2024-06-06T03:09:53.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:53 smithi153 ceph-mon[29300]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:53.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:53 smithi153 ceph-mon[29300]: Saving service mon spec with placement count:5 2024-06-06T03:09:53.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:53 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:55.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:54 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:55.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:54 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:55.184 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:54 smithi153 ceph-mon[29300]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:55.185 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:54 smithi153 ceph-mon[29300]: Saving service mgr spec with placement count:2 2024-06-06T03:09:55.185 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:54 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.387+0000 7fb8f74b1640 1 Processor -- start 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f74b1640 1 -- start start 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f74b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f74b1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f0078140 con 0x7fb8f0077d30 2024-06-06T03:09:55.192 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f524d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f524d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47696/0 (socket says 172.21.15.153:47696) 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f524d640 1 -- 172.21.15.153:0/110453588 learned_addr learned my addr 172.21.15.153:0/110453588 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.388+0000 7fb8f524d640 1 -- 172.21.15.153:0/110453588 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f00767a0 con 0x7fb8f0077d30 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.389+0000 7fb8f524d640 1 --2- 172.21.15.153:0/110453588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009840 tx=0x7fb8ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69118b9ed9862b55 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.389+0000 7fb8e7fff640 1 -- 172.21.15.153:0/110453588 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8ec032440 con 0x7fb8f0077d30 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.389+0000 7fb8e7fff640 1 -- 172.21.15.153:0/110453588 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8ec032a70 con 0x7fb8f0077d30 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.389+0000 7fb8e7fff640 1 -- 172.21.15.153:0/110453588 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8ec031ad0 con 0x7fb8f0077d30 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 -- 172.21.15.153:0/110453588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 msgr2=0x7fb8f0076260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/110453588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009840 tx=0x7fb8ec0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 -- 172.21.15.153:0/110453588 shutdown_connections 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/110453588 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f0076260 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 -- 172.21.15.153:0/110453588 >> 172.21.15.153:0/110453588 conn(0x7fb8f00fd470 msgr2=0x7fb8f00ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:55.193 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 -- 172.21.15.153:0/110453588 shutdown_connections 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.390+0000 7fb8f74b1640 1 -- 172.21.15.153:0/110453588 wait complete. 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f74b1640 1 Processor -- start 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f74b1640 1 -- start start 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f74b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f74b1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f01135f0 con 0x7fb8f0077d30 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f524d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f524d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47702/0 (socket says 172.21.15.153:47702) 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f524d640 1 -- 172.21.15.153:0/794274640 learned_addr learned my addr 172.21.15.153:0/794274640 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.391+0000 7fb8f524d640 1 -- 172.21.15.153:0/794274640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ec009450 con 0x7fb8f0077d30 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8f524d640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0317f0 tx=0x7fb8ec02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8ec032900 con 0x7fb8f0077d30 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8ec03b910 con 0x7fb8f0077d30 2024-06-06T03:09:55.194 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f01137f0 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8ec03a960 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.392+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f0113c60 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.393+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb8ec039070 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.393+0000 7fb8e5ffb640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 0x7fb8dc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.393+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f004b480 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.393+0000 7fb8f4a4c640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 0x7fb8dc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.393+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb8ec066c70 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.394+0000 7fb8f4a4c640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 0x7fb8dc036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.398+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb8ec04c350 con 0x7fb8f0077d30 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.582+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fb8f0113c60 con 0x7fb8dc034810 2024-06-06T03:09:55.195 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.586+0000 7fb8e5ffb640 1 -- 172.21.15.153:0/794274640 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb8f0113c60 con 0x7fb8dc034810 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 msgr2=0x7fb8dc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 0x7fb8dc036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb8e0006fd0 tx=0x7fb8e0006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 msgr2=0x7fb8f01130b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec0317f0 tx=0x7fb8ec02c040 comp rx=0 tx=0).stop 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 shutdown_connections 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fb8dc034810 0x7fb8dc036cc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 --2- 172.21.15.153:0/794274640 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f0077d30 0x7fb8f01130b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 >> 172.21.15.153:0/794274640 conn(0x7fb8f00fd470 msgr2=0x7fb8f00fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 shutdown_connections 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:54.588+0000 7fb8f74b1640 1 -- 172.21.15.153:0/794274640 wait complete. 2024-06-06T03:09:55.196 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying crash service with default placement... 2024-06-06T03:09:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:56 smithi153 ceph-mon[29300]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:09:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:56 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:56.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-06T03:09:56.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c312b640 1 Processor -- start 2024-06-06T03:09:56.801 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c312b640 1 -- start start 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c312b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c312b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc100ad0 con 0x7ff6bc100180 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c0ec7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c0ec7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47704/0 (socket says 172.21.15.153:47704) 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.605+0000 7ff6c0ec7640 1 -- 172.21.15.153:0/4103459396 learned_addr learned my addr 172.21.15.153:0/4103459396 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.606+0000 7ff6c0ec7640 1 -- 172.21.15.153:0/4103459396 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc100c10 con 0x7ff6bc100180 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.606+0000 7ff6c0ec7640 1 --2- 172.21.15.153:0/4103459396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009840 tx=0x7ff6b00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b03a1056a731bec server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.606+0000 7ff6bb7fe640 1 -- 172.21.15.153:0/4103459396 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6b0032440 con 0x7ff6bc100180 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6bb7fe640 1 -- 172.21.15.153:0/4103459396 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6b0032a70 con 0x7ff6bc100180 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6bb7fe640 1 -- 172.21.15.153:0/4103459396 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6b0031ad0 con 0x7ff6bc100180 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 -- 172.21.15.153:0/4103459396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 msgr2=0x7ff6bc100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4103459396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009840 tx=0x7ff6b00091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:56.802 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 -- 172.21.15.153:0/4103459396 shutdown_connections 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4103459396 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc100590 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 -- 172.21.15.153:0/4103459396 >> 172.21.15.153:0/4103459396 conn(0x7ff6bc0fbbc0 msgr2=0x7ff6bc0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 -- 172.21.15.153:0/4103459396 shutdown_connections 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.607+0000 7ff6c312b640 1 -- 172.21.15.153:0/4103459396 wait complete. 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c312b640 1 Processor -- start 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c312b640 1 -- start start 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c312b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c312b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc1110e0 con 0x7ff6bc100180 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c0ec7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c0ec7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47706/0 (socket says 172.21.15.153:47706) 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c0ec7640 1 -- 172.21.15.153:0/4164252729 learned_addr learned my addr 172.21.15.153:0/4164252729 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.608+0000 7ff6c0ec7640 1 -- 172.21.15.153:0/4164252729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b0009450 con 0x7ff6bc100180 2024-06-06T03:09:56.803 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6c0ec7640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009810 tx=0x7ff6b002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6b00328b0 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff6b003b910 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6b003a980 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc1112e0 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.609+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc111750 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff6b0039070 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6b97fa640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 0x7ff6ac036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc04b480 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff6b0066de0 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6bbfff640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 0x7ff6ac036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.610+0000 7ff6bbfff640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 0x7ff6ac036d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8006fd0 tx=0x7ff6a8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.614+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff6b0037070 con 0x7ff6bc100180 2024-06-06T03:09:56.804 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.792+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6bc0fcf70 con 0x7ff6ac034860 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.796+0000 7ff6b97fa640 1 -- 172.21.15.153:0/4164252729 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7ff6bc0fcf70 con 0x7ff6ac034860 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 msgr2=0x7ff6ac036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 0x7ff6ac036d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff6a8006fd0 tx=0x7ff6a8006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 msgr2=0x7ff6bc110ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009810 tx=0x7ff6b002c040 comp rx=0 tx=0).stop 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 shutdown_connections 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7ff6ac034860 0x7ff6ac036d10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 --2- 172.21.15.153:0/4164252729 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff6bc100180 0x7ff6bc110ba0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 >> 172.21.15.153:0/4164252729 conn(0x7ff6bc0fbbc0 msgr2=0x7ff6bc0fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 shutdown_connections 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:55.798+0000 7ff6c312b640 1 -- 172.21.15.153:0/4164252729 wait complete. 2024-06-06T03:09:56.805 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying prometheus service with default placement... 2024-06-06T03:09:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:57 smithi153 ceph-mon[29300]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:57 smithi153 ceph-mon[29300]: Saving service crash spec with placement * 2024-06-06T03:09:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:57 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:57 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:57 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8ad09cf640 1 Processor -- start 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8ad09cf640 1 -- start start 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8ad09cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8ad09cf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc1015a0 con 0x7f8acc100c50 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8aca59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:58.035 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8aca59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47712/0 (socket says 172.21.15.153:47712) 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8aca59c640 1 -- 172.21.15.153:0/3774060517 learned_addr learned my addr 172.21.15.153:0/3774060517 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.167+0000 7f8aca59c640 1 -- 172.21.15.153:0/3774060517 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc1016e0 con 0x7f8acc100c50 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.168+0000 7f8aca59c640 1 --2- 172.21.15.153:0/3774060517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009840 tx=0x7f8abc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ff0be1fcce3b471 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.168+0000 7f8ac9d9b640 1 -- 172.21.15.153:0/3774060517 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc032440 con 0x7f8acc100c50 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.168+0000 7f8ac9d9b640 1 -- 172.21.15.153:0/3774060517 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8abc032a70 con 0x7f8acc100c50 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.168+0000 7f8ac9d9b640 1 -- 172.21.15.153:0/3774060517 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc031ad0 con 0x7f8acc100c50 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3774060517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 msgr2=0x7f8acc101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3774060517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f8abc009840 tx=0x7f8abc0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3774060517 shutdown_connections 2024-06-06T03:09:58.036 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3774060517 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc101060 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3774060517 >> 172.21.15.153:0/3774060517 conn(0x7f8acc0fc430 msgr2=0x7f8acc0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3774060517 shutdown_connections 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.169+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3774060517 wait complete. 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8ad09cf640 1 Processor -- start 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8ad09cf640 1 -- start start 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8ad09cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8ad09cf640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8acc111130 con 0x7f8acc100c50 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8aca59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8aca59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47726/0 (socket says 172.21.15.153:47726) 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8aca59c640 1 -- 172.21.15.153:0/3526385960 learned_addr learned my addr 172.21.15.153:0/3526385960 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8aca59c640 1 -- 172.21.15.153:0/3526385960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc009450 con 0x7f8acc100c50 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.170+0000 7f8aca59c640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8abc006a70 tx=0x7f8abc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:58.037 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc031e90 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8acc111330 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8abc03b910 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc03a940 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8acc1117a0 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.171+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f8abc039070 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.172+0000 7f8ac2ffd640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 0x7f8ab8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.172+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f8abc066ca0 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.172+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8acc05ead0 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.172+0000 7f8ac3fff640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 0x7f8ab8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.172+0000 7f8ac3fff640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 0x7f8ab8036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006fd0 tx=0x7f8ab4006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.176+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8abc03f080 con 0x7f8acc100c50 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.374+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f8acc10a910 con 0x7f8ab8034860 2024-06-06T03:09:58.038 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.379+0000 7f8ac2ffd640 1 -- 172.21.15.153:0/3526385960 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f8acc10a910 con 0x7f8ab8034860 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 msgr2=0x7f8ab8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 0x7f8ab8036d10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4006fd0 tx=0x7f8ab4006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 msgr2=0x7f8acc110bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8abc006a70 tx=0x7f8abc02c040 comp rx=0 tx=0).stop 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 shutdown_connections 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f8ab8034860 0x7f8ab8036d10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 --2- 172.21.15.153:0/3526385960 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8acc100c50 0x7f8acc110bf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 >> 172.21.15.153:0/3526385960 conn(0x7f8acc0fc430 msgr2=0x7f8acc0fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 shutdown_connections 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:57.381+0000 7f8ad09cf640 1 -- 172.21.15.153:0/3526385960 wait complete. 2024-06-06T03:09:58.039 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying grafana service with default placement... 2024-06-06T03:09:58.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:58 smithi153 ceph-mon[29300]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:09:58.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:58 smithi153 ceph-mon[29300]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:58.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:58 smithi153 ceph-mon[29300]: Saving service prometheus spec with placement count:1 2024-06-06T03:09:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-06T03:09:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.412+0000 7f3b30a6d640 1 Processor -- start 2024-06-06T03:09:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.412+0000 7f3b30a6d640 1 -- start start 2024-06-06T03:09:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.412+0000 7f3b30a6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:59.568 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.412+0000 7f3b30a6d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c100a30 con 0x7f3b2c100100 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.413+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.413+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47742/0 (socket says 172.21.15.153:47742) 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.413+0000 7f3b2a59c640 1 -- 172.21.15.153:0/670123303 learned_addr learned my addr 172.21.15.153:0/670123303 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.413+0000 7f3b2a59c640 1 -- 172.21.15.153:0/670123303 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c100b70 con 0x7f3b2c100100 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b2a59c640 1 --2- 172.21.15.153:0/670123303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009840 tx=0x7f3b1c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aff43af656c7f06e server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b29d9b640 1 -- 172.21.15.153:0/670123303 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c032440 con 0x7f3b2c100100 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b29d9b640 1 -- 172.21.15.153:0/670123303 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b1c032a70 con 0x7f3b2c100100 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b29d9b640 1 -- 172.21.15.153:0/670123303 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c031ad0 con 0x7f3b2c100100 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 -- 172.21.15.153:0/670123303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 msgr2=0x7f3b2c1004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/670123303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c009840 tx=0x7f3b1c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 -- 172.21.15.153:0/670123303 shutdown_connections 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/670123303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c1004f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:59.569 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 -- 172.21.15.153:0/670123303 >> 172.21.15.153:0/670123303 conn(0x7f3b2c0fbbc0 msgr2=0x7f3b2c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 -- 172.21.15.153:0/670123303 shutdown_connections 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.414+0000 7f3b30a6d640 1 -- 172.21.15.153:0/670123303 wait complete. 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b30a6d640 1 Processor -- start 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b30a6d640 1 -- start start 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b30a6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b30a6d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c1110a0 con 0x7f3b2c100100 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b2a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47754/0 (socket says 172.21.15.153:47754) 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.415+0000 7f3b2a59c640 1 -- 172.21.15.153:0/3784949031 learned_addr learned my addr 172.21.15.153:0/3784949031 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b2a59c640 1 -- 172.21.15.153:0/3784949031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c009450 con 0x7f3b2c100100 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b2a59c640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c006a70 tx=0x7f3b1c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:59.570 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c031e90 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b1c03b910 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b1c03a940 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b2c1112a0 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.416+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2c111710 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3b1c039070 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b237fe640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 0x7f3b18036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3b1c066db0 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b2c05ead0 con 0x7f3b2c100100 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b21d9b640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 0x7f3b18036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.417+0000 7f3b21d9b640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 0x7f3b18036cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006fd0 tx=0x7f3b14006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:09:59.571 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.422+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3b1c03f080 con 0x7f3b2c100100 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.599+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b2c0fcec0 con 0x7f3b18034810 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.603+0000 7f3b237fe640 1 -- 172.21.15.153:0/3784949031 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f3b2c0fcec0 con 0x7f3b18034810 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 msgr2=0x7f3b18036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 0x7f3b18036cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3b14006fd0 tx=0x7f3b14006db0 comp rx=0 tx=0).stop 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 msgr2=0x7f3b2c110b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3b1c006a70 tx=0x7f3b1c02c040 comp rx=0 tx=0).stop 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 shutdown_connections 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f3b18034810 0x7f3b18036cc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 --2- 172.21.15.153:0/3784949031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b2c100100 0x7f3b2c110b60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 >> 172.21.15.153:0/3784949031 conn(0x7f3b2c0fbbc0 msgr2=0x7f3b2c0fc770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 shutdown_connections 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:58.605+0000 7f3b30a6d640 1 -- 172.21.15.153:0/3784949031 wait complete. 2024-06-06T03:09:59.572 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying node-exporter service with default placement... 2024-06-06T03:09:59.864 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:59 smithi153 ceph-mon[29300]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:09:59.864 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:59 smithi153 ceph-mon[29300]: Saving service grafana spec with placement count:1 2024-06-06T03:09:59.864 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:09:59 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:00.828 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff416f640 1 Processor -- start 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff416f640 1 -- start start 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff416f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff416f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec104ea0 con 0x7f5fec075bb0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff1f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff1f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47768/0 (socket says 172.21.15.153:47768) 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.941+0000 7f5ff1f0b640 1 -- 172.21.15.153:0/2285389941 learned_addr learned my addr 172.21.15.153:0/2285389941 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.942+0000 7f5ff1f0b640 1 -- 172.21.15.153:0/2285389941 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec104fe0 con 0x7f5fec075bb0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.942+0000 7f5ff1f0b640 1 --2- 172.21.15.153:0/2285389941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009840 tx=0x7f5fe40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db6501d41b25b676 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.942+0000 7f5ff0f09640 1 -- 172.21.15.153:0/2285389941 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe4032440 con 0x7f5fec075bb0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.942+0000 7f5ff0f09640 1 -- 172.21.15.153:0/2285389941 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fe4032a70 con 0x7f5fec075bb0 2024-06-06T03:10:00.829 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.942+0000 7f5ff0f09640 1 -- 172.21.15.153:0/2285389941 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe4031ad0 con 0x7f5fec075bb0 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 -- 172.21.15.153:0/2285389941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 msgr2=0x7f5fec104960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 --2- 172.21.15.153:0/2285389941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4009840 tx=0x7f5fe40091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 -- 172.21.15.153:0/2285389941 shutdown_connections 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 --2- 172.21.15.153:0/2285389941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec104960 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 -- 172.21.15.153:0/2285389941 >> 172.21.15.153:0/2285389941 conn(0x7f5fec0fc470 msgr2=0x7f5fec0fe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 -- 172.21.15.153:0/2285389941 shutdown_connections 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.943+0000 7f5ff416f640 1 -- 172.21.15.153:0/2285389941 wait complete. 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff416f640 1 Processor -- start 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff416f640 1 -- start start 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff416f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff416f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec1027d0 con 0x7f5fec075bb0 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff1f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff1f0b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47770/0 (socket says 172.21.15.153:47770) 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff1f0b640 1 -- 172.21.15.153:0/808555827 learned_addr learned my addr 172.21.15.153:0/808555827 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff1f0b640 1 -- 172.21.15.153:0/808555827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe4009450 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff1f0b640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40317f0 tx=0x7f5fe402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe4032900 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fe403b910 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.944+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec100860 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.945+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fe403a960 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.945+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec100cd0 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.945+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f5fe4039070 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.945+0000 7f5fea7fc640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 0x7f5fd4036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.945+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5fe4066dc0 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.946+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec04b480 con 0x7f5fec075bb0 2024-06-06T03:10:00.831 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.946+0000 7f5ff170a640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 0x7f5fd4036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.946+0000 7f5ff170a640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 0x7f5fd4036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0006fd0 tx=0x7f5fe0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:09:59.950+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5fe403f080 con 0x7f5fec075bb0 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.129+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fec101250 con 0x7f5fd4034810 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.133+0000 7f5fea7fc640 1 -- 172.21.15.153:0/808555827 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f5fec101250 con 0x7f5fd4034810 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 msgr2=0x7f5fd4036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 0x7f5fd4036cc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0006fd0 tx=0x7f5fe0006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 msgr2=0x7f5fec102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5fe40317f0 tx=0x7f5fe402c040 comp rx=0 tx=0).stop 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 shutdown_connections 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f5fd4034810 0x7f5fd4036cc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 --2- 172.21.15.153:0/808555827 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5fec075bb0 0x7f5fec102290 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 >> 172.21.15.153:0/808555827 conn(0x7f5fec0fc470 msgr2=0x7f5fec0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 shutdown_connections 2024-06-06T03:10:00.833 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:00.135+0000 7f5ff416f640 1 -- 172.21.15.153:0/808555827 wait complete. 2024-06-06T03:10:00.833 INFO:teuthology.orchestra.run.smithi153.stdout:Deploying alertmanager service with default placement... 2024-06-06T03:10:00.928 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:00 smithi153 ceph-mon[29300]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:00.929 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:00 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:02.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:01 smithi153 ceph-mon[29300]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:02.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:01 smithi153 ceph-mon[29300]: Saving service node-exporter spec with placement * 2024-06-06T03:10:02.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:01 smithi153 ceph-mon[29300]: from='mgr.14118 172.21.15.153:0/2737557944' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116ee6e640 1 Processor -- start 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116ee6e640 1 -- start start 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116ee6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116ee6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168102600 con 0x7f1168101cb0 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116cc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116cc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49266/0 (socket says 172.21.15.153:49266) 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116cc0a640 1 -- 172.21.15.153:0/1794132061 learned_addr learned my addr 172.21.15.153:0/1794132061 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.176+0000 7f116cc0a640 1 -- 172.21.15.153:0/1794132061 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1168102740 con 0x7f1168101cb0 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.177+0000 7f116cc0a640 1 --2- 172.21.15.153:0/1794132061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f115c009840 tx=0x7f115c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3665c547cfe865 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.177+0000 7f11677fe640 1 -- 172.21.15.153:0/1794132061 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c032440 con 0x7f1168101cb0 2024-06-06T03:10:02.009 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.177+0000 7f11677fe640 1 -- 172.21.15.153:0/1794132061 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f115c032a70 con 0x7f1168101cb0 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.177+0000 7f11677fe640 1 -- 172.21.15.153:0/1794132061 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c031ad0 con 0x7f1168101cb0 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 -- 172.21.15.153:0/1794132061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 msgr2=0x7f11681020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 --2- 172.21.15.153:0/1794132061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f115c009840 tx=0x7f115c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 -- 172.21.15.153:0/1794132061 shutdown_connections 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 --2- 172.21.15.153:0/1794132061 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f11681020c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 -- 172.21.15.153:0/1794132061 >> 172.21.15.153:0/1794132061 conn(0x7f11680fd490 msgr2=0x7f11680ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 -- 172.21.15.153:0/1794132061 shutdown_connections 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 -- 172.21.15.153:0/1794132061 wait complete. 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.178+0000 7f116ee6e640 1 Processor -- start 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116ee6e640 1 -- start start 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116ee6e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116ee6e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11681113e0 con 0x7f1168101cb0 2024-06-06T03:10:02.010 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116cc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116cc0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49276/0 (socket says 172.21.15.153:49276) 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116cc0a640 1 -- 172.21.15.153:0/200295281 learned_addr learned my addr 172.21.15.153:0/200295281 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116cc0a640 1 -- 172.21.15.153:0/200295281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c009450 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f116cc0a640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f115c006a70 tx=0x7f115c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.179+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c031e90 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.180+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f115c03b910 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.180+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11681115e0 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.180+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f115c03a940 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.180+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168111a50 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.180+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f115c039070 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.181+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f116804b480 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.181+0000 7f11657fa640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 0x7f1158036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.181+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f115c066c30 con 0x7f1168101cb0 2024-06-06T03:10:02.011 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.181+0000 7f1167fff640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 0x7f1158036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.181+0000 7f1167fff640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 0x7f1158036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1154006fd0 tx=0x7f1154006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.185+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f115c039320 con 0x7f1168101cb0 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.360+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f1168111a50 con 0x7f1158034860 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.364+0000 7f11657fa640 1 -- 172.21.15.153:0/200295281 <== mgr.14118 v2:172.21.15.153:6800/1921588694 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f1168111a50 con 0x7f1158034860 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 msgr2=0x7f1158036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 0x7f1158036d10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1154006fd0 tx=0x7f1154006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 msgr2=0x7f1168110ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f115c006a70 tx=0x7f115c02c040 comp rx=0 tx=0).stop 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 shutdown_connections 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f1158034860 0x7f1158036d10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 --2- 172.21.15.153:0/200295281 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1168101cb0 0x7f1168110ea0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:02.012 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 >> 172.21.15.153:0/200295281 conn(0x7f11680fd490 msgr2=0x7f11680fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:02.013 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 shutdown_connections 2024-06-06T03:10:02.013 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:01.366+0000 7f116ee6e640 1 -- 172.21.15.153:0/200295281 wait complete. 2024-06-06T03:10:02.924 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:02 smithi153 ceph-mon[29300]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:02.924 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:02 smithi153 ceph-mon[29300]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:02.924 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:02 smithi153 ceph-mon[29300]: Saving service alertmanager spec with placement count:1 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.440+0000 7f8104332640 1 Processor -- start 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.440+0000 7f8104332640 1 -- start start 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f8104332640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f8104332640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc102700 con 0x7f80fc101db0 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f81020ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f81020ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49286/0 (socket says 172.21.15.153:49286) 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f81020ce640 1 -- 172.21.15.153:0/500194749 learned_addr learned my addr 172.21.15.153:0/500194749 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.441+0000 7f81020ce640 1 -- 172.21.15.153:0/500194749 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc102840 con 0x7f80fc101db0 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.442+0000 7f81020ce640 1 --2- 172.21.15.153:0/500194749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009840 tx=0x7f80f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f495c6af57fc943 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.442+0000 7f81010cc640 1 -- 172.21.15.153:0/500194749 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f0032440 con 0x7f80fc101db0 2024-06-06T03:10:03.199 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.442+0000 7f81010cc640 1 -- 172.21.15.153:0/500194749 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80f0032a70 con 0x7f80fc101db0 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.442+0000 7f81010cc640 1 -- 172.21.15.153:0/500194749 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f0031ad0 con 0x7f80fc101db0 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- 172.21.15.153:0/500194749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 msgr2=0x7f80fc1021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 --2- 172.21.15.153:0/500194749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009840 tx=0x7f80f00091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- 172.21.15.153:0/500194749 shutdown_connections 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 --2- 172.21.15.153:0/500194749 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc1021c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- 172.21.15.153:0/500194749 >> 172.21.15.153:0/500194749 conn(0x7f80fc0fd5b0 msgr2=0x7f80fc0ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- 172.21.15.153:0/500194749 shutdown_connections 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- 172.21.15.153:0/500194749 wait complete. 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 Processor -- start 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.443+0000 7f8104332640 1 -- start start 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f8104332640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f8104332640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc111080 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f81020ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f81020ce640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49302/0 (socket says 172.21.15.153:49302) 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f81020ce640 1 -- 172.21.15.153:0/1407760547 learned_addr learned my addr 172.21.15.153:0/1407760547 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f81020ce640 1 -- 172.21.15.153:0/1407760547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0009450 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f81020ce640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009810 tx=0x7f80f002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f00328b0 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80f003b910 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80fc111280 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80f003a980 con 0x7f80fc101db0 2024-06-06T03:10:03.201 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.444+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80fc1116f0 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.445+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f80f0039070 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.445+0000 7f80eeffd640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 0x7f80e8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.445+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80fc04b480 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.446+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f80f0066e30 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.446+0000 7f81018cd640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 0x7f80e8036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.446+0000 7f81018cd640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 0x7f80e8036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.450+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f80f004c350 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.614+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f80fc103b40 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.619+0000 7f80eeffd640 1 -- 172.21.15.153:0/1407760547 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f80f0037020 con 0x7f80fc101db0 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 msgr2=0x7f80e8036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 0x7f80e8036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f80f8006fd0 tx=0x7f80f8006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 msgr2=0x7f80fc110b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:03.202 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009810 tx=0x7f80f002c040 comp rx=0 tx=0).stop 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 shutdown_connections 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f80e8034860 0x7f80e8036d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 --2- 172.21.15.153:0/1407760547 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80fc101db0 0x7f80fc110b40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 >> 172.21.15.153:0/1407760547 conn(0x7f80fc0fd5b0 msgr2=0x7f80fc0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 shutdown_connections 2024-06-06T03:10:03.203 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:02.622+0000 7f8104332640 1 -- 172.21.15.153:0/1407760547 wait complete. 2024-06-06T03:10:03.777 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:03 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1407760547' entity='client.admin' 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.603+0000 7fd83ad59640 1 Processor -- start 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.603+0000 7fd83ad59640 1 -- start start 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.603+0000 7fd83ad59640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.603+0000 7fd83ad59640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834075f00 con 0x7fd834075af0 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.603+0000 7fd838af5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.604+0000 7fd838af5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49314/0 (socket says 172.21.15.153:49314) 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.604+0000 7fd838af5640 1 -- 172.21.15.153:0/3788565595 learned_addr learned my addr 172.21.15.153:0/3788565595 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.604+0000 7fd838af5640 1 -- 172.21.15.153:0/3788565595 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8341027f0 con 0x7fd834075af0 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.604+0000 7fd838af5640 1 --2- 172.21.15.153:0/3788565595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd830009840 tx=0x7fd8300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff5b0a3166b7b8 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd82b7fe640 1 -- 172.21.15.153:0/3788565595 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd830032440 con 0x7fd834075af0 2024-06-06T03:10:04.305 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd82b7fe640 1 -- 172.21.15.153:0/3788565595 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd830032a70 con 0x7fd834075af0 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd82b7fe640 1 -- 172.21.15.153:0/3788565595 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd830031ad0 con 0x7fd834075af0 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 -- 172.21.15.153:0/3788565595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 msgr2=0x7fd834106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3788565595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd830009840 tx=0x7fd8300091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 -- 172.21.15.153:0/3788565595 shutdown_connections 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3788565595 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834106e00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 -- 172.21.15.153:0/3788565595 >> 172.21.15.153:0/3788565595 conn(0x7fd8340fc430 msgr2=0x7fd8340fe8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 -- 172.21.15.153:0/3788565595 shutdown_connections 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.605+0000 7fd83ad59640 1 -- 172.21.15.153:0/3788565595 wait complete. 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd83ad59640 1 Processor -- start 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd83ad59640 1 -- start start 2024-06-06T03:10:04.306 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd83ad59640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd83ad59640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd834113340 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd838af5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd838af5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49324/0 (socket says 172.21.15.153:49324) 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.606+0000 7fd838af5640 1 -- 172.21.15.153:0/3721402189 learned_addr learned my addr 172.21.15.153:0/3721402189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd838af5640 1 -- 172.21.15.153:0/3721402189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd830009450 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd838af5640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8300098a0 tx=0x7fd83002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd830031e90 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd83003b910 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8341135a0 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd83003a9b0 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.607+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd834113f90 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.608+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fd830039070 con 0x7fd834075af0 2024-06-06T03:10:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.608+0000 7fd8297fa640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 0x7fd81c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.608+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd830066f50 con 0x7fd834075af0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.608+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83405ead0 con 0x7fd834075af0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.608+0000 7fd82bfff640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 0x7fd81c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.609+0000 7fd82bfff640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 0x7fd81c036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd824006fd0 tx=0x7fd824006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.613+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd830037070 con 0x7fd834075af0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.778+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd83404b480 con 0x7fd834075af0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.783+0000 7fd8297fa640 1 -- 172.21.15.153:0/3721402189 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd830030930 con 0x7fd834075af0 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 msgr2=0x7fd81c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 0x7fd81c036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd824006fd0 tx=0x7fd824006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 msgr2=0x7fd834112e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8300098a0 tx=0x7fd83002c040 comp rx=0 tx=0).stop 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 shutdown_connections 2024-06-06T03:10:04.308 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7fd81c034860 0x7fd81c036d10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:04.310 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 --2- 172.21.15.153:0/3721402189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd834075af0 0x7fd834112e00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:04.310 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 >> 172.21.15.153:0/3721402189 conn(0x7fd8340fc430 msgr2=0x7fd8340fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:04.310 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 shutdown_connections 2024-06-06T03:10:04.310 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:03.786+0000 7fd83ad59640 1 -- 172.21.15.153:0/3721402189 wait complete. 2024-06-06T03:10:04.310 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling the dashboard module... 2024-06-06T03:10:04.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:04 smithi153 ceph-mon[29300]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:04.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:04 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3721402189' entity='client.admin' 2024-06-06T03:10:05.901 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:05 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1940336085' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-06T03:10:06.269 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.713+0000 7f7e4c86d640 1 Processor -- start 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4c86d640 1 -- start start 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4c86d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4c86d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e48102660 con 0x7f7e48101d10 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49334/0 (socket says 172.21.15.153:49334) 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4659c640 1 -- 172.21.15.153:0/2550510744 learned_addr learned my addr 172.21.15.153:0/2550510744 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.714+0000 7f7e4659c640 1 -- 172.21.15.153:0/2550510744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e481027a0 con 0x7f7e48101d10 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.715+0000 7f7e4659c640 1 --2- 172.21.15.153:0/2550510744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009840 tx=0x7f7e380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4160006d0636388 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.715+0000 7f7e4559a640 1 -- 172.21.15.153:0/2550510744 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38032440 con 0x7f7e48101d10 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.715+0000 7f7e4559a640 1 -- 172.21.15.153:0/2550510744 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e38032a70 con 0x7f7e48101d10 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.715+0000 7f7e4559a640 1 -- 172.21.15.153:0/2550510744 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e38031ad0 con 0x7f7e48101d10 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- 172.21.15.153:0/2550510744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 msgr2=0x7f7e48102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:06.270 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/2550510744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009840 tx=0x7f7e380091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- 172.21.15.153:0/2550510744 shutdown_connections 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/2550510744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48102120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- 172.21.15.153:0/2550510744 >> 172.21.15.153:0/2550510744 conn(0x7f7e480fd4f0 msgr2=0x7f7e480ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- 172.21.15.153:0/2550510744 shutdown_connections 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- 172.21.15.153:0/2550510744 wait complete. 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 Processor -- start 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- start start 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.716+0000 7f7e4c86d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e48078150 con 0x7f7e48101d10 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49346/0 (socket says 172.21.15.153:49346) 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4659c640 1 -- 172.21.15.153:0/1940336085 learned_addr learned my addr 172.21.15.153:0/1940336085 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:06.271 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4659c640 1 -- 172.21.15.153:0/1940336085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e38009450 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4659c640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009810 tx=0x7f7e3802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e380328b0 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e3803b910 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e480761e0 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e3803a980 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.717+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e48076650 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.718+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f7e38039070 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.718+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4805ead0 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.719+0000 7f7e3effd640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 0x7f7e34036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.719+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f7e38066f00 con 0x7f7e48101d10 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.719+0000 7f7e45d9b640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 0x7f7e34036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.719+0000 7f7e45d9b640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 0x7f7e34036d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006fd0 tx=0x7f7e30006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:06.272 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.723+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7e38037450 con 0x7f7e48101d10 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:04.931+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f7e4804b480 con 0x7f7e48101d10 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.628+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f7e3804c350 con 0x7f7e48101d10 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.629+0000 7f7e3effd640 1 -- 172.21.15.153:0/1940336085 <== mon.0 v2:172.21.15.153:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f7e3803aae0 con 0x7f7e48101d10 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.631+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 msgr2=0x7f7e34036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 0x7f7e34036d10 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7e30006fd0 tx=0x7f7e30006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 msgr2=0x7f7e48077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7e38009810 tx=0x7f7e3802c040 comp rx=0 tx=0).stop 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 shutdown_connections 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f7e34034860 0x7f7e34036d10 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 --2- 172.21.15.153:0/1940336085 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e48101d10 0x7f7e48077c10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 >> 172.21.15.153:0/1940336085 conn(0x7f7e480fd4f0 msgr2=0x7f7e480fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 shutdown_connections 2024-06-06T03:10:06.273 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:05.632+0000 7f7e4c86d640 1 -- 172.21.15.153:0/1940336085 wait complete. 2024-06-06T03:10:06.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:06 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1940336085' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-06T03:10:06.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:06 smithi153 ceph-mon[29300]: mgrmap e9: smithi153.gidtue(active, since 30s) 2024-06-06T03:10:07.727 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:10:07.727 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-06T03:10:07.727 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-06T03:10:07.727 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "active_name": "smithi153.gidtue", 2024-06-06T03:10:07.727 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.658+0000 7f11cf144640 1 Processor -- start 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.658+0000 7f11cf144640 1 -- start start 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.658+0000 7f11cf144640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.658+0000 7f11cf144640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c8075f70 con 0x7f11c8075b60 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.658+0000 7f11ccee0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.659+0000 7f11ccee0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49380/0 (socket says 172.21.15.153:49380) 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.659+0000 7f11ccee0640 1 -- 172.21.15.153:0/1067103438 learned_addr learned my addr 172.21.15.153:0/1067103438 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.659+0000 7f11ccee0640 1 -- 172.21.15.153:0/1067103438 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c8104ec0 con 0x7f11c8075b60 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.659+0000 7f11ccee0640 1 --2- 172.21.15.153:0/1067103438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009840 tx=0x7f11bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cca2f6908b63799 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11c77fe640 1 -- 172.21.15.153:0/1067103438 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11bc032440 con 0x7f11c8075b60 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11c77fe640 1 -- 172.21.15.153:0/1067103438 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11bc032a70 con 0x7f11c8075b60 2024-06-06T03:10:07.728 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11c77fe640 1 -- 172.21.15.153:0/1067103438 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11bc031ad0 con 0x7f11c8075b60 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 -- 172.21.15.153:0/1067103438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 msgr2=0x7f11c8104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 --2- 172.21.15.153:0/1067103438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009840 tx=0x7f11bc0091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 -- 172.21.15.153:0/1067103438 shutdown_connections 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 --2- 172.21.15.153:0/1067103438 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8104980 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 -- 172.21.15.153:0/1067103438 >> 172.21.15.153:0/1067103438 conn(0x7f11c80fc430 msgr2=0x7f11c80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 -- 172.21.15.153:0/1067103438 shutdown_connections 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.660+0000 7f11cf144640 1 -- 172.21.15.153:0/1067103438 wait complete. 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11cf144640 1 Processor -- start 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11cf144640 1 -- start start 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11cf144640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11cf144640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c8111110 con 0x7f11c8075b60 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11ccee0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11ccee0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49394/0 (socket says 172.21.15.153:49394) 2024-06-06T03:10:07.729 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.661+0000 7f11ccee0640 1 -- 172.21.15.153:0/3220548113 learned_addr learned my addr 172.21.15.153:0/3220548113 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11ccee0640 1 -- 172.21.15.153:0/3220548113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc009450 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11ccee0640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0098a0 tx=0x7f11bc02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11bc031e90 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11bc03b910 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11bc03a9b0 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11c8111310 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.662+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c8111780 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f11bc039070 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11c57fa640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 0x7f11b8036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f11bc066ec0 con 0x7f11c8075b60 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11c7fff640 1 -- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 msgr2=0x7f11b8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11c7fff640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 0x7f11b8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T03:10:07.730 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.663+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c805ead0 con 0x7f11c8075b60 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.668+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f11bc03f080 con 0x7f11c8075b60 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.864+0000 7f11c7fff640 1 -- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 msgr2=0x7f11b8036d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.864+0000 7f11c7fff640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 0x7f11b8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.879+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f11c8111fa0 con 0x7f11c8075b60 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.879+0000 7f11c57fa640 1 -- 172.21.15.153:0/3220548113 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7f11bc037070 con 0x7f11c8075b60 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.882+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 msgr2=0x7f11b8036d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.882+0000 7f11cf144640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 0x7f11b8036d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.882+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 msgr2=0x7f11c8110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.882+0000 7f11cf144640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f11bc0098a0 tx=0x7f11bc02c040 comp rx=0 tx=0).stop 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 shutdown_connections 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f11b8034860 0x7f11b8036d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:07.731 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 --2- 172.21.15.153:0/3220548113 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11c8075b60 0x7f11c8110bd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:07.732 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 >> 172.21.15.153:0/3220548113 conn(0x7f11c80fc430 msgr2=0x7f11c80fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:07.732 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 shutdown_connections 2024-06-06T03:10:07.732 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:06.883+0000 7f11cf144640 1 -- 172.21.15.153:0/3220548113 wait complete. 2024-06-06T03:10:07.732 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for the mgr to restart... 2024-06-06T03:10:07.732 INFO:teuthology.orchestra.run.smithi153.stdout:Waiting for mgr epoch 9... 2024-06-06T03:10:07.872 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:07 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3220548113' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-06T03:10:10.397 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: Active manager daemon smithi153.gidtue restarted 2024-06-06T03:10:10.397 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: Activating manager daemon smithi153.gidtue 2024-06-06T03:10:10.397 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: osdmap e3: 0 total, 0 up, 0 in 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: mgrmap e10: smithi153.gidtue(active, starting, since 0.00576115s) 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr metadata", "who": "smithi153.gidtue", "id": "smithi153.gidtue"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: Manager daemon smithi153.gidtue is now available 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/mirror_snapshot_schedule"}]: dispatch 2024-06-06T03:10:10.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:10 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/trash_purge_schedule"}]: dispatch 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout { 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout } 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.092+0000 7f44d9837640 1 Processor -- start 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d9837640 1 -- start start 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d9837640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d9837640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d4102600 con 0x7f44d4101cb0 2024-06-06T03:10:12.155 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49408/0 (socket says 172.21.15.153:49408) 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d2ffd640 1 -- 172.21.15.153:0/3261562288 learned_addr learned my addr 172.21.15.153:0/3261562288 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.093+0000 7f44d2ffd640 1 -- 172.21.15.153:0/3261562288 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d4102740 con 0x7f44d4101cb0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.094+0000 7f44d2ffd640 1 --2- 172.21.15.153:0/3261562288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009840 tx=0x7f44c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b930130121c4296 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.094+0000 7f44d1ffb640 1 -- 172.21.15.153:0/3261562288 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c4032440 con 0x7f44d4101cb0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.094+0000 7f44d1ffb640 1 -- 172.21.15.153:0/3261562288 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44c4032a70 con 0x7f44d4101cb0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.094+0000 7f44d1ffb640 1 -- 172.21.15.153:0/3261562288 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c4031ad0 con 0x7f44d4101cb0 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 -- 172.21.15.153:0/3261562288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 msgr2=0x7f44d41020c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 --2- 172.21.15.153:0/3261562288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009840 tx=0x7f44c40091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 -- 172.21.15.153:0/3261562288 shutdown_connections 2024-06-06T03:10:12.156 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 --2- 172.21.15.153:0/3261562288 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d41020c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 -- 172.21.15.153:0/3261562288 >> 172.21.15.153:0/3261562288 conn(0x7f44d40fd490 msgr2=0x7f44d40ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 -- 172.21.15.153:0/3261562288 shutdown_connections 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.095+0000 7f44d9837640 1 -- 172.21.15.153:0/3261562288 wait complete. 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d9837640 1 Processor -- start 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d9837640 1 -- start start 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d9837640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d9837640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d4111100 con 0x7f44d4101cb0 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49418/0 (socket says 172.21.15.153:49418) 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.096+0000 7f44d2ffd640 1 -- 172.21.15.153:0/2039632351 learned_addr learned my addr 172.21.15.153:0/2039632351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44d2ffd640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c4009450 con 0x7f44d4101cb0 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44d2ffd640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009810 tx=0x7f44c402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:12.157 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c40328b0 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f44c403b910 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d4111300 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44c403a980 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.097+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d4111770 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f44c4039070 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44bb7fe640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44d405ead0 con 0x7f44bc0348b0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f44c4067d60 con 0x7f44d4101cb0 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44d27fc640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 msgr2=0x7f44bc036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.098+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.299+0000 7f44d27fc640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 msgr2=0x7f44bc036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.299+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T03:10:12.158 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.700+0000 7f44d27fc640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 msgr2=0x7f44bc036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:08.700+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:09.501+0000 7f44d27fc640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 msgr2=0x7f44bc036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/1921588694 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:09.501+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:10.225+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7f44c403adb0 con 0x7f44d4101cb0 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:10.225+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 msgr2=0x7f44bc036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:10.225+0000 7f44bb7fe640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.223+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f44c4040260 con 0x7f44d4101cb0 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.223+0000 7f44bb7fe640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 0x7f44bc03a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.223+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f44d405ead0 con 0x7f44bc037fe0 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.224+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 0x7f44bc03a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.224+0000 7f44d27fc640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 0x7f44bc03a3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44c8003940 tx=0x7f44c8009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:12.159 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.225+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f44d405ead0 con 0x7f44bc037fe0 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.230+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f44d404b480 con 0x7f44bc037fe0 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.230+0000 7f44bb7fe640 1 -- 172.21.15.153:0/2039632351 <== mgr.14160 v2:172.21.15.153:6800/659590756 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f44d404b480 con 0x7f44bc037fe0 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 msgr2=0x7f44bc03a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 0x7f44bc03a3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f44c8003940 tx=0x7f44c8009350 comp rx=0 tx=0).stop 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 msgr2=0x7f44d4110bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f44c4009810 tx=0x7f44c402c040 comp rx=0 tx=0).stop 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 shutdown_connections 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f44bc037fe0 0x7f44bc03a3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:6800/1921588694,v1:172.21.15.153:6801/1921588694] conn(0x7f44bc0348b0 0x7f44bc036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 --2- 172.21.15.153:0/2039632351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f44d4101cb0 0x7f44d4110bc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 >> 172.21.15.153:0/2039632351 conn(0x7f44d40fd490 msgr2=0x7f44d40fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 shutdown_connections 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:11.231+0000 7f44d9837640 1 -- 172.21.15.153:0/2039632351 wait complete. 2024-06-06T03:10:12.160 INFO:teuthology.orchestra.run.smithi153.stdout:mgr epoch 9 is available 2024-06-06T03:10:12.161 INFO:teuthology.orchestra.run.smithi153.stdout:Generating a dashboard self-signed certificate... 2024-06-06T03:10:12.482 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: mgrmap e11: smithi153.gidtue(active, since 1.00534s) 2024-06-06T03:10:12.482 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:10:11] ENGINE Bus STARTING 2024-06-06T03:10:12.482 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:12.482 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:10:11] ENGINE Serving on https://172.21.15.153:7150 2024-06-06T03:10:12.482 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:10:11] ENGINE Bus STARTED 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:10:11] ENGINE Error in HTTPServer.serve 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Traceback (most recent call last): 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._run(expiration_interval) 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s = self.context.wrap_socket( 2024-06-06T03:10:12.483 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: return self.sslsocket_class._create( 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self.do_handshake() 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._sslobj.do_handshake() 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T03:10:12.484 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:12 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:13.318 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-06T03:10:13.318 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.559+0000 7f9c6733e640 1 Processor -- start 2024-06-06T03:10:13.318 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.559+0000 7f9c6733e640 1 -- start start 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.559+0000 7f9c6733e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.559+0000 7f9c6733e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c60104e80 con 0x7f9c60075ba0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.560+0000 7f9c650da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.560+0000 7f9c650da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46652/0 (socket says 172.21.15.153:46652) 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.560+0000 7f9c650da640 1 -- 172.21.15.153:0/2231438610 learned_addr learned my addr 172.21.15.153:0/2231438610 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.560+0000 7f9c650da640 1 -- 172.21.15.153:0/2231438610 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c60104fc0 con 0x7f9c60075ba0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c650da640 1 --2- 172.21.15.153:0/2231438610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009840 tx=0x7f9c5c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc310c437f97f8f server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c57fff640 1 -- 172.21.15.153:0/2231438610 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5c032440 con 0x7f9c60075ba0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c57fff640 1 -- 172.21.15.153:0/2231438610 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c5c032a70 con 0x7f9c60075ba0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c57fff640 1 -- 172.21.15.153:0/2231438610 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5c031ad0 con 0x7f9c60075ba0 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 -- 172.21.15.153:0/2231438610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 msgr2=0x7f9c60104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:13.319 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2231438610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009840 tx=0x7f9c5c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 -- 172.21.15.153:0/2231438610 shutdown_connections 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2231438610 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60104940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 -- 172.21.15.153:0/2231438610 >> 172.21.15.153:0/2231438610 conn(0x7f9c600fc430 msgr2=0x7f9c600fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.561+0000 7f9c6733e640 1 -- 172.21.15.153:0/2231438610 shutdown_connections 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c6733e640 1 -- 172.21.15.153:0/2231438610 wait complete. 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c6733e640 1 Processor -- start 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c6733e640 1 -- start start 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c6733e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c6733e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c601027b0 con 0x7f9c60075ba0 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c650da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c650da640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46656/0 (socket says 172.21.15.153:46656) 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c650da640 1 -- 172.21.15.153:0/2864432159 learned_addr learned my addr 172.21.15.153:0/2864432159 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:13.320 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.562+0000 7f9c650da640 1 -- 172.21.15.153:0/2864432159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c009450 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c650da640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0317f0 tx=0x7f9c5c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5c032900 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c5c03b910 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c60100840 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c5c03a960 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.563+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c60100cb0 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.564+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7f9c5c039070 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.564+0000 7f9c55ffb640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 0x7f9c4c036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.564+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c6004b480 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.564+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9c5c066da0 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.564+0000 7f9c648d9640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 0x7f9c4c036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.565+0000 7f9c648d9640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 0x7f9c4c036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9c50006fd0 tx=0x7f9c50006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.569+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c5c03ba80 con 0x7f9c60075ba0 2024-06-06T03:10:13.321 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.638+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f9c5c03b440 con 0x7f9c60075ba0 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.747+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c60101460 con 0x7f9c4c034790 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.790+0000 7f9c55ffb640 1 -- 172.21.15.153:0/2864432159 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f9c60101460 con 0x7f9c4c034790 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 msgr2=0x7f9c4c036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 0x7f9c4c036c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9c50006fd0 tx=0x7f9c50006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 msgr2=0x7f9c60102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c0317f0 tx=0x7f9c5c02c040 comp rx=0 tx=0).stop 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 shutdown_connections 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9c4c034790 0x7f9c4c036c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 --2- 172.21.15.153:0/2864432159 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c60075ba0 0x7f9c60102270 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 >> 172.21.15.153:0/2864432159 conn(0x7f9c600fc430 msgr2=0x7f9c600fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 shutdown_connections 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:12.792+0000 7f9c6733e640 1 -- 172.21.15.153:0/2864432159 wait complete. 2024-06-06T03:10:13.322 INFO:teuthology.orchestra.run.smithi153.stdout:Creating initial admin user... 2024-06-06T03:10:13.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:13 smithi153 ceph-mon[29300]: mgrmap e12: smithi153.gidtue(active, since 2s) 2024-06-06T03:10:13.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:13 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:13.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:13 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:13.918 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:13 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:14.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:14 smithi153 ceph-mon[29300]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:14.838 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:14 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:15.108 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$R7Cztjnpe5zSUkZBJtMtqu9tJdqqD2e5.ibVM9Euxto1pV/kVxoIy", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1717643414, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-06T03:10:15.175 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.721+0000 7fc144626640 1 Processor -- start 2024-06-06T03:10:15.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.721+0000 7fc144626640 1 -- start start 2024-06-06T03:10:15.182 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc144626640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc144626640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c075f50 con 0x7fc13c075b40 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc1423c2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc1423c2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46670/0 (socket says 172.21.15.153:46670) 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc1423c2640 1 -- 172.21.15.153:0/2187296490 learned_addr learned my addr 172.21.15.153:0/2187296490 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.722+0000 7fc1423c2640 1 -- 172.21.15.153:0/2187296490 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c104ee0 con 0x7fc13c075b40 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc1423c2640 1 --2- 172.21.15.153:0/2187296490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc130009840 tx=0x7fc1300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69074d5498b04be7 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc1413c0640 1 -- 172.21.15.153:0/2187296490 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc130032440 con 0x7fc13c075b40 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc1413c0640 1 -- 172.21.15.153:0/2187296490 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc130032a70 con 0x7fc13c075b40 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc1413c0640 1 -- 172.21.15.153:0/2187296490 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc130031ad0 con 0x7fc13c075b40 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc144626640 1 -- 172.21.15.153:0/2187296490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 msgr2=0x7fc13c1049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.723+0000 7fc144626640 1 --2- 172.21.15.153:0/2187296490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc130009840 tx=0x7fc1300091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- 172.21.15.153:0/2187296490 shutdown_connections 2024-06-06T03:10:15.183 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 --2- 172.21.15.153:0/2187296490 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1049a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- 172.21.15.153:0/2187296490 >> 172.21.15.153:0/2187296490 conn(0x7fc13c0fc430 msgr2=0x7fc13c0fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- 172.21.15.153:0/2187296490 shutdown_connections 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- 172.21.15.153:0/2187296490 wait complete. 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 Processor -- start 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- start start 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.724+0000 7fc144626640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c102810 con 0x7fc13c075b40 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc1423c2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc1423c2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46678/0 (socket says 172.21.15.153:46678) 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc1423c2640 1 -- 172.21.15.153:0/3808085552 learned_addr learned my addr 172.21.15.153:0/3808085552 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc1423c2640 1 -- 172.21.15.153:0/3808085552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc130009450 con 0x7fc13c075b40 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc1423c2640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1300098a0 tx=0x7fc13002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc130031e90 con 0x7fc13c075b40 2024-06-06T03:10:15.184 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc13003b910 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c1008a0 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc13003a9b0 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.725+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c100d10 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.726+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fc130039070 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.726+0000 7fc12effd640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 0x7fc128036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.726+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c04b480 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.726+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc1300671a0 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.726+0000 7fc141bc1640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 0x7fc128036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.727+0000 7fc141bc1640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 0x7fc128036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc138006fd0 tx=0x7fc138006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.731+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc130037070 con 0x7fc13c075b40 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:13.918+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fc13c05e8c0 con 0x7fc1280348b0 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.141+0000 7fc12effd640 1 -- 172.21.15.153:0/3808085552 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fc13c05e8c0 con 0x7fc1280348b0 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 msgr2=0x7fc128036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:15.185 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 0x7fc128036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc138006fd0 tx=0x7fc138006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 msgr2=0x7fc13c1022d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc1300098a0 tx=0x7fc13002c040 comp rx=0 tx=0).stop 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 shutdown_connections 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fc1280348b0 0x7fc128036d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 --2- 172.21.15.153:0/3808085552 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc13c075b40 0x7fc13c1022d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 >> 172.21.15.153:0/3808085552 conn(0x7fc13c0fc430 msgr2=0x7fc13c0fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 shutdown_connections 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:14.143+0000 7fc144626640 1 -- 172.21.15.153:0/3808085552 wait complete. 2024-06-06T03:10:15.186 INFO:teuthology.orchestra.run.smithi153.stdout:Fetching dashboard port number... 2024-06-06T03:10:15.935 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:15 smithi153 ceph-mon[29300]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:16.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stdout 8443 2024-06-06T03:10:16.521 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.550+0000 7f9c19fa8640 1 Processor -- start 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.550+0000 7f9c19fa8640 1 -- start start 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.550+0000 7f9c19fa8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.550+0000 7f9c19fa8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c141015a0 con 0x7f9c14100c50 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.551+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.551+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46686/0 (socket says 172.21.15.153:46686) 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.551+0000 7f9c137fe640 1 -- 172.21.15.153:0/1795792175 learned_addr learned my addr 172.21.15.153:0/1795792175 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.551+0000 7f9c137fe640 1 -- 172.21.15.153:0/1795792175 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c141016e0 con 0x7f9c14100c50 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c137fe640 1 --2- 172.21.15.153:0/1795792175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009e50 tx=0x7f9c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66592ca9b7a356b9 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c127fc640 1 -- 172.21.15.153:0/1795792175 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c04031560 con 0x7f9c14100c50 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c127fc640 1 -- 172.21.15.153:0/1795792175 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c0402d050 con 0x7f9c14100c50 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c127fc640 1 -- 172.21.15.153:0/1795792175 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c04007ad0 con 0x7f9c14100c50 2024-06-06T03:10:16.522 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1795792175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 msgr2=0x7f9c14101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1795792175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009e50 tx=0x7f9c04009840 comp rx=0 tx=0).stop 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1795792175 shutdown_connections 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1795792175 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14101060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1795792175 >> 172.21.15.153:0/1795792175 conn(0x7f9c140fc430 msgr2=0x7f9c140fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1795792175 shutdown_connections 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.552+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1795792175 wait complete. 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c19fa8640 1 Processor -- start 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c19fa8640 1 -- start start 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c19fa8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c19fa8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14113910 con 0x7f9c14100c50 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46688/0 (socket says 172.21.15.153:46688) 2024-06-06T03:10:16.523 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.553+0000 7f9c137fe640 1 -- 172.21.15.153:0/1813630076 learned_addr learned my addr 172.21.15.153:0/1813630076 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9c137fe640 1 -- 172.21.15.153:0/1813630076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04009ae0 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9c137fe640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c040077b0 tx=0x7f9c04009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c04031de0 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c0403d920 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c14113a50 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c0403b6e0 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.554+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14113ec0 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.555+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f9c0403b840 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.555+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1404b480 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.555+0000 7f9bfbfff640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 0x7f9bfc036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.555+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9c04067840 con 0x7f9c14100c50 2024-06-06T03:10:16.524 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.555+0000 7f9c12ffd640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 0x7f9bfc036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.556+0000 7f9c12ffd640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 0x7f9bfc036d60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c08006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.560+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c040446d0 con 0x7f9c14100c50 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.729+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f9c14114050 con 0x7f9c14100c50 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.730+0000 7f9bfbfff640 1 -- 172.21.15.153:0/1813630076 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f9c04041070 con 0x7f9c14100c50 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 msgr2=0x7f9bfc036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 0x7f9bfc036d60 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9c08006fd0 tx=0x7f9c08006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 msgr2=0x7f9c14075af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9c040077b0 tx=0x7f9c04009840 comp rx=0 tx=0).stop 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 shutdown_connections 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f9bfc0348b0 0x7f9bfc036d60 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 --2- 172.21.15.153:0/1813630076 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c14100c50 0x7f9c14075af0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 >> 172.21.15.153:0/1813630076 conn(0x7f9c140fc430 msgr2=0x7f9c140fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:16.525 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 shutdown_connections 2024-06-06T03:10:16.526 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:15.732+0000 7f9c19fa8640 1 -- 172.21.15.153:0/1813630076 wait complete. 2024-06-06T03:10:16.531 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-06T03:10:16.531 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout disabled 2024-06-06T03:10:16.540 INFO:teuthology.orchestra.run.smithi153.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-06T03:10:16.540 INFO:teuthology.orchestra.run.smithi153.stdout:systemctl: stdout inactive 2024-06-06T03:10:16.540 INFO:teuthology.orchestra.run.smithi153.stdout:firewalld.service is not enabled 2024-06-06T03:10:16.540 INFO:teuthology.orchestra.run.smithi153.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout:Ceph Dashboard is now available at: 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: URL: https://smithi153.front.sepia.ceph.com:8443/ 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: User: admin 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: Password: yobdc2gw03 2024-06-06T03:10:16.542 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:16.543 INFO:teuthology.orchestra.run.smithi153.stdout:Saving cluster configuration to /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config directory 2024-06-06T03:10:16.543 INFO:teuthology.orchestra.run.smithi153.stdout:Enabling autotune for osd_memory_target 2024-06-06T03:10:16.930 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:16 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1813630076' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.930+0000 7f19a93ac640 1 Processor -- start 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.930+0000 7f19a93ac640 1 -- start start 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.930+0000 7f19a93ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.930+0000 7f19a93ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4102700 con 0x7f19a4101db0 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.931+0000 7f19a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.931+0000 7f19a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46702/0 (socket says 172.21.15.153:46702) 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.931+0000 7f19a2ffd640 1 -- 172.21.15.153:0/3734360672 learned_addr learned my addr 172.21.15.153:0/3734360672 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.931+0000 7f19a2ffd640 1 -- 172.21.15.153:0/3734360672 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4102840 con 0x7f19a4101db0 2024-06-06T03:10:17.645 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a2ffd640 1 --2- 172.21.15.153:0/3734360672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1994009840 tx=0x7f19940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7eeb77072e2f597 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a1ffb640 1 -- 172.21.15.153:0/3734360672 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1994032440 con 0x7f19a4101db0 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a1ffb640 1 -- 172.21.15.153:0/3734360672 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1994032a70 con 0x7f19a4101db0 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a1ffb640 1 -- 172.21.15.153:0/3734360672 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1994031ad0 con 0x7f19a4101db0 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 -- 172.21.15.153:0/3734360672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 msgr2=0x7f19a41021c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3734360672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f1994009840 tx=0x7f19940091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 -- 172.21.15.153:0/3734360672 shutdown_connections 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3734360672 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a41021c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 -- 172.21.15.153:0/3734360672 >> 172.21.15.153:0/3734360672 conn(0x7f19a40fd5b0 msgr2=0x7f19a40ff9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 -- 172.21.15.153:0/3734360672 shutdown_connections 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.932+0000 7f19a93ac640 1 -- 172.21.15.153:0/3734360672 wait complete. 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.933+0000 7f19a93ac640 1 Processor -- start 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.933+0000 7f19a93ac640 1 -- start start 2024-06-06T03:10:17.646 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.933+0000 7f19a93ac640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.933+0000 7f19a93ac640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4078150 con 0x7f19a4101db0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46706/0 (socket says 172.21.15.153:46706) 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a2ffd640 1 -- 172.21.15.153:0/3108186781 learned_addr learned my addr 172.21.15.153:0/3108186781 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a2ffd640 1 -- 172.21.15.153:0/3108186781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994009450 con 0x7f19a4101db0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a2ffd640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1994009810 tx=0x7f199402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19940328b0 con 0x7f19a4101db0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f199403b910 con 0x7f19a4101db0 2024-06-06T03:10:17.647 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a40761e0 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f199403a980 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.934+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a4076650 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.935+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f1994039070 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.935+0000 7f198b7fe640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 0x7f198c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.935+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1994068110 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.936+0000 7f19a27fc640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 0x7f198c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.936+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a404b480 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.936+0000 7f19a27fc640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 0x7f198c036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f1998006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:16.940+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f199403f080 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.104+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f19a4076b80 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.105+0000 7f198b7fe640 1 -- 172.21.15.153:0/3108186781 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f1994037070 con 0x7f19a4101db0 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.106+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 msgr2=0x7f198c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:17.648 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.106+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 0x7f198c036db0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1998006fd0 tx=0x7f1998006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.106+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 msgr2=0x7f19a4077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.106+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1994009810 tx=0x7f199402c040 comp rx=0 tx=0).stop 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 shutdown_connections 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f198c034900 0x7f198c036db0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 --2- 172.21.15.153:0/3108186781 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f19a4101db0 0x7f19a4077c10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 >> 172.21.15.153:0/3108186781 conn(0x7f19a40fd5b0 msgr2=0x7f19a40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 shutdown_connections 2024-06-06T03:10:17.649 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:17.107+0000 7f19a93ac640 1 -- 172.21.15.153:0/3108186781 wait complete. 2024-06-06T03:10:17.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:17 smithi153 ceph-mon[29300]: mgrmap e13: smithi153.gidtue(active, since 6s) 2024-06-06T03:10:18.876 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd842aa640 1 Processor -- start 2024-06-06T03:10:18.876 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd842aa640 1 -- start start 2024-06-06T03:10:18.876 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd842aa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:18.876 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd842aa640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c102660 con 0x7fcd7c101d10 2024-06-06T03:10:18.876 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd82046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd82046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46720/0 (socket says 172.21.15.153:46720) 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd82046640 1 -- 172.21.15.153:0/1281149759 learned_addr learned my addr 172.21.15.153:0/1281149759 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.033+0000 7fcd82046640 1 -- 172.21.15.153:0/1281149759 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7c1027a0 con 0x7fcd7c101d10 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.034+0000 7fcd82046640 1 --2- 172.21.15.153:0/1281149759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009840 tx=0x7fcd780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530865b6cc7b67ab server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.034+0000 7fcd81044640 1 -- 172.21.15.153:0/1281149759 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd78032440 con 0x7fcd7c101d10 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.034+0000 7fcd81044640 1 -- 172.21.15.153:0/1281149759 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd78032a70 con 0x7fcd7c101d10 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.034+0000 7fcd81044640 1 -- 172.21.15.153:0/1281149759 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd78031ad0 con 0x7fcd7c101d10 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 -- 172.21.15.153:0/1281149759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 msgr2=0x7fcd7c102120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 --2- 172.21.15.153:0/1281149759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009840 tx=0x7fcd780091d0 comp rx=0 tx=0).stop 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 -- 172.21.15.153:0/1281149759 shutdown_connections 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 --2- 172.21.15.153:0/1281149759 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c102120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:18.877 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 -- 172.21.15.153:0/1281149759 >> 172.21.15.153:0/1281149759 conn(0x7fcd7c0fd550 msgr2=0x7fcd7c0ff960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 -- 172.21.15.153:0/1281149759 shutdown_connections 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.035+0000 7fcd842aa640 1 -- 172.21.15.153:0/1281149759 wait complete. 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd842aa640 1 Processor -- start 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd842aa640 1 -- start start 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd842aa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd842aa640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7c078150 con 0x7fcd7c101d10 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd82046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd82046640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46722/0 (socket says 172.21.15.153:46722) 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd82046640 1 -- 172.21.15.153:0/3546048612 learned_addr learned my addr 172.21.15.153:0/3546048612 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd82046640 1 -- 172.21.15.153:0/3546048612 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd78009450 con 0x7fcd7c101d10 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd82046640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009810 tx=0x7fcd7802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd780328b0 con 0x7fcd7c101d10 2024-06-06T03:10:18.878 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcd7803b910 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd7803a980 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.036+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7c0761e0 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.037+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7c076650 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.037+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fcd78039070 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.037+0000 7fcd72ffd640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 0x7fcd68036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.037+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fcd780671c0 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.037+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7c04b480 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.038+0000 7fcd81845640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 0x7fcd68036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.038+0000 7fcd81845640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 0x7fcd68036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c006fd0 tx=0x7fcd6c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.042+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd78037070 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.288+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fcd7c0769b0 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.290+0000 7fcd72ffd640 1 -- 172.21.15.153:0/3546048612 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7fcd78030930 con 0x7fcd7c101d10 2024-06-06T03:10:18.879 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 msgr2=0x7fcd68036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 0x7fcd68036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c006fd0 tx=0x7fcd6c006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 msgr2=0x7fcd7c077c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcd78009810 tx=0x7fcd7802c040 comp rx=0 tx=0).stop 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 shutdown_connections 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fcd68034950 0x7fcd68036e00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 --2- 172.21.15.153:0/3546048612 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcd7c101d10 0x7fcd7c077c10 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 >> 172.21.15.153:0/3546048612 conn(0x7fcd7c0fd550 msgr2=0x7fcd7c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 shutdown_connections 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:/usr/bin/ceph: stderr 2024-06-06T03:10:18.292+0000 7fcd842aa640 1 -- 172.21.15.153:0/3546048612 wait complete. 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.880 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout:Or, if you are only running a single cluster on this host: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: ceph telemetry on 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout:For more information see: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-06T03:10:18.881 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:18.882 INFO:teuthology.orchestra.run.smithi153.stdout:Bootstrap complete. 2024-06-06T03:10:18.909 INFO:tasks.cephadm:Fetching config... 2024-06-06T03:10:18.909 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:10:18.909 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-06T03:10:18.925 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-06T03:10:18.925 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:10:18.925 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-06T03:10:18.980 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-06T03:10:18.981 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:10:18.981 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/keyring of=/dev/stdout 2024-06-06T03:10:19.045 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-06T03:10:19.046 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:10:19.046 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-06T03:10:19.101 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-06T03:10:19.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4yiWuyIIQ+5WRiIVB23IHvsfHZwbjWTQdQ2/EEhL8tKfekI5YiQJXqjm4QCY96RwuqJXin72ifSnnUXEbyzJYW3VcBTlPnGgEPmltcjC2fRF6w68GJoo3fYgxqYv50qHEaOx1IhLifrL5/IxSGe/1bj4jmKYVjlclg4IvsEhfzzTnZls0j+MIOPZSz7bWAdy3WNdkG5zlCN49m34mHTaZ+Z5QDtqsd/2oZ49AR/QyN7CTXcnBzMo4dr0K15BHVS9c511ffSYFecrLvjjZwhitdahmpaQMB7NsAQbe5GsebN4++o2gL9BLegcIrZfYODXugq0NnR0hWyxGtoKfzifW2BaDZQZmEQ8fihIudZ4uTR7v7LGNJn/yB+/ashDcuK3B2ABxkVJR/qIZScaCLNy9ZIlsJHNFyt3D0fwEcMVdIWZVgXGzmCC8enrL6zIa4B6x69EBTKLRvaLuBtz33E25N95QnttP+N/hCdiZ54EzQZZ83tJTZrm8lpxn97jRCYE= ceph-0b80117e-23b2-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T03:10:19.177 INFO:teuthology.orchestra.run.smithi153.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4yiWuyIIQ+5WRiIVB23IHvsfHZwbjWTQdQ2/EEhL8tKfekI5YiQJXqjm4QCY96RwuqJXin72ifSnnUXEbyzJYW3VcBTlPnGgEPmltcjC2fRF6w68GJoo3fYgxqYv50qHEaOx1IhLifrL5/IxSGe/1bj4jmKYVjlclg4IvsEhfzzTnZls0j+MIOPZSz7bWAdy3WNdkG5zlCN49m34mHTaZ+Z5QDtqsd/2oZ49AR/QyN7CTXcnBzMo4dr0K15BHVS9c511ffSYFecrLvjjZwhitdahmpaQMB7NsAQbe5GsebN4++o2gL9BLegcIrZfYODXugq0NnR0hWyxGtoKfzifW2BaDZQZmEQ8fihIudZ4uTR7v7LGNJn/yB+/ashDcuK3B2ABxkVJR/qIZScaCLNy9ZIlsJHNFyt3D0fwEcMVdIWZVgXGzmCC8enrL6zIa4B6x69EBTKLRvaLuBtz33E25N95QnttP+N/hCdiZ54EzQZZ83tJTZrm8lpxn97jRCYE= ceph-0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:10:19.189 DEBUG:teuthology.orchestra.run.smithi173:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4yiWuyIIQ+5WRiIVB23IHvsfHZwbjWTQdQ2/EEhL8tKfekI5YiQJXqjm4QCY96RwuqJXin72ifSnnUXEbyzJYW3VcBTlPnGgEPmltcjC2fRF6w68GJoo3fYgxqYv50qHEaOx1IhLifrL5/IxSGe/1bj4jmKYVjlclg4IvsEhfzzTnZls0j+MIOPZSz7bWAdy3WNdkG5zlCN49m34mHTaZ+Z5QDtqsd/2oZ49AR/QyN7CTXcnBzMo4dr0K15BHVS9c511ffSYFecrLvjjZwhitdahmpaQMB7NsAQbe5GsebN4++o2gL9BLegcIrZfYODXugq0NnR0hWyxGtoKfzifW2BaDZQZmEQ8fihIudZ4uTR7v7LGNJn/yB+/ashDcuK3B2ABxkVJR/qIZScaCLNy9ZIlsJHNFyt3D0fwEcMVdIWZVgXGzmCC8enrL6zIa4B6x69EBTKLRvaLuBtz33E25N95QnttP+N/hCdiZ54EzQZZ83tJTZrm8lpxn97jRCYE= ceph-0b80117e-23b2-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-06T03:10:19.236 INFO:teuthology.orchestra.run.smithi173.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4yiWuyIIQ+5WRiIVB23IHvsfHZwbjWTQdQ2/EEhL8tKfekI5YiQJXqjm4QCY96RwuqJXin72ifSnnUXEbyzJYW3VcBTlPnGgEPmltcjC2fRF6w68GJoo3fYgxqYv50qHEaOx1IhLifrL5/IxSGe/1bj4jmKYVjlclg4IvsEhfzzTnZls0j+MIOPZSz7bWAdy3WNdkG5zlCN49m34mHTaZ+Z5QDtqsd/2oZ49AR/QyN7CTXcnBzMo4dr0K15BHVS9c511ffSYFecrLvjjZwhitdahmpaQMB7NsAQbe5GsebN4++o2gL9BLegcIrZfYODXugq0NnR0hWyxGtoKfzifW2BaDZQZmEQ8fihIudZ4uTR7v7LGNJn/yB+/ashDcuK3B2ABxkVJR/qIZScaCLNy9ZIlsJHNFyt3D0fwEcMVdIWZVgXGzmCC8enrL6zIa4B6x69EBTKLRvaLuBtz33E25N95QnttP+N/hCdiZ54EzQZZ83tJTZrm8lpxn97jRCYE= ceph-0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:10:19.249 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-06T03:10:19.500 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:10:19.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:19 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3546048612' entity='client.admin' 2024-06-06T03:10:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 -- 172.21.15.153:0/878204474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 msgr2=0x7f80100fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 --2- 172.21.15.153:0/878204474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f80100fc320 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8000004850 tx=0x7f8000009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 -- 172.21.15.153:0/878204474 shutdown_connections 2024-06-06T03:10:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 --2- 172.21.15.153:0/878204474 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f80100fc320 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 -- 172.21.15.153:0/878204474 >> 172.21.15.153:0/878204474 conn(0x7f80100f76f0 msgr2=0x7f80100f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 -- 172.21.15.153:0/878204474 shutdown_connections 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 -- 172.21.15.153:0/878204474 wait complete. 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.870+0000 7f80166c4640 1 Processor -- start 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f80166c4640 1 -- start start 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f80166c4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f80166c4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801010ca50 con 0x7f80100fbf10 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f800ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f800ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39012/0 (socket says 172.21.15.153:39012) 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f800ffff640 1 -- 172.21.15.153:0/564946504 learned_addr learned my addr 172.21.15.153:0/564946504 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.871+0000 7f800ffff640 1 -- 172.21.15.153:0/564946504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000005040 con 0x7f80100fbf10 2024-06-06T03:10:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f800ffff640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f80000098d0 tx=0x7f8000009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8000032650 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8000032c80 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801010cc50 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8000033c10 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.872+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801010d0c0 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.873+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f8000030020 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.873+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801005ead0 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.873+0000 7f800cff9640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.873+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8000066620 con 0x7f80100fbf10 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.873+0000 7f800f7fe640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.874+0000 7f800f7fe640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 0x7f7ff8036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8004006fd0 tx=0x7f8004006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:21.877+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f800003acd0 con 0x7f80100fbf10 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:10:21.972 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T03:10:21.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:10:21.973 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:21 smithi153 ceph-mon[29300]: Deploying daemon crash.smithi153 on smithi153 2024-06-06T03:10:22.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.049+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f801010d0c0 con 0x7f80100fbf10 2024-06-06T03:10:22.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.054+0000 7f800cff9640 1 -- 172.21.15.153:0/564946504 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f800003a8e0 con 0x7f80100fbf10 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 msgr2=0x7f7ff8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 0x7f7ff8036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8004006fd0 tx=0x7f8004006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 msgr2=0x7f801010c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f80000098d0 tx=0x7f8000009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 shutdown_connections 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7ff8034900 0x7f7ff8036db0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 --2- 172.21.15.153:0/564946504 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f80100fbf10 0x7f801010c510 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 >> 172.21.15.153:0/564946504 conn(0x7f80100f76f0 msgr2=0x7f80101030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 shutdown_connections 2024-06-06T03:10:22.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:22.056+0000 7f80166c4640 1 -- 172.21.15.153:0/564946504 wait complete. 2024-06-06T03:10:22.697 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-06T03:10:22.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-06T03:10:23.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:23 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/564946504' entity='client.admin' 2024-06-06T03:10:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 -- 172.21.15.153:0/4268167285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 msgr2=0x7f7e8c0fb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 --2- 172.21.15.153:0/4268167285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0fb8c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7e88004850 tx=0x7f7e88009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 -- 172.21.15.153:0/4268167285 shutdown_connections 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 --2- 172.21.15.153:0/4268167285 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0fb8c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 -- 172.21.15.153:0/4268167285 >> 172.21.15.153:0/4268167285 conn(0x7f7e8c0f5da0 msgr2=0x7f7e8c0f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 -- 172.21.15.153:0/4268167285 shutdown_connections 2024-06-06T03:10:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.696+0000 7f7e93972640 1 -- 172.21.15.153:0/4268167285 wait complete. 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e93972640 1 Processor -- start 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e93972640 1 -- start start 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e93972640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e93972640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c10d040 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e9170e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e9170e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39024/0 (socket says 172.21.15.153:39024) 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e9170e640 1 -- 172.21.15.153:0/1020140760 learned_addr learned my addr 172.21.15.153:0/1020140760 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.697+0000 7f7e9170e640 1 -- 172.21.15.153:0/1020140760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e88005040 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e9170e640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009f80 tx=0x7f7e88009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e880326c0 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c111650 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7e88032cf0 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e8803dd40 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.698+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c111980 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f7e88030020 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c05ead0 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e81ffb640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 0x7f7e78036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7e88066f80 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e90f0d640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 0x7f7e78036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.699+0000 7f7e90f0d640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 0x7f7e78036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.704+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7e88045390 con 0x7f7e8c0fb4b0 2024-06-06T03:10:25.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.877+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f7e8c05d940 con 0x7f7e78034900 2024-06-06T03:10:25.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.880+0000 7f7e81ffb640 1 -- 172.21.15.153:0/1020140760 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7e8c05d940 con 0x7f7e78034900 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 msgr2=0x7f7e78036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 0x7f7e78036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c006fd0 tx=0x7f7e7c006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 msgr2=0x7f7e8c0facd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e88009f80 tx=0x7f7e88009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 shutdown_connections 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f7e78034900 0x7f7e78036db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 --2- 172.21.15.153:0/1020140760 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e8c0fb4b0 0x7f7e8c0facd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 >> 172.21.15.153:0/1020140760 conn(0x7f7e8c0f5da0 msgr2=0x7f7e8c101ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 shutdown_connections 2024-06-06T03:10:25.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:25.882+0000 7f7e93972640 1 -- 172.21.15.153:0/1020140760 wait complete. 2024-06-06T03:10:26.528 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi173 2024-06-06T03:10:26.529 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:10:26.529 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.conf 2024-06-06T03:10:26.547 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:10:26.548 DEBUG:teuthology.orchestra.run.smithi173:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T03:10:26.605 INFO:tasks.cephadm:Adding host smithi173 to orchestrator... 2024-06-06T03:10:26.606 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch host add smithi173 2024-06-06T03:10:26.882 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:27.135 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:26 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:28.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: Deploying daemon node-exporter.smithi153 on smithi153 2024-06-06T03:10:28.137 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='client.14186 -' entity='client.crash.smithi153' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:28.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:28.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='client.14188 -' entity='client.crash.smithi153' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:28.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:28.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='client.14190 -' entity='client.crash.smithi153' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:28.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:28 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:28.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.469+0000 7f000196e640 1 -- 172.21.15.153:0/1196635840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 msgr2=0x7efffc0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:28.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.469+0000 7f000196e640 1 --2- 172.21.15.153:0/1196635840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc0fc320 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7effec004850 tx=0x7effec009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:28.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- 172.21.15.153:0/1196635840 shutdown_connections 2024-06-06T03:10:28.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 --2- 172.21.15.153:0/1196635840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc0fc320 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- 172.21.15.153:0/1196635840 >> 172.21.15.153:0/1196635840 conn(0x7efffc0f76f0 msgr2=0x7efffc0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- 172.21.15.153:0/1196635840 shutdown_connections 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- 172.21.15.153:0/1196635840 wait complete. 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 Processor -- start 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- start start 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7f000196e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efffc10ca50 con 0x7efffc0fbf10 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.470+0000 7efffaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efffaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39120/0 (socket says 172.21.15.153:39120) 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efffaffd640 1 -- 172.21.15.153:0/1883925158 learned_addr learned my addr 172.21.15.153:0/1883925158 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:28.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efffaffd640 1 -- 172.21.15.153:0/1883925158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec005040 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efffaffd640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7effec004820 tx=0x7effec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effec032650 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effec032c80 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efffc10cc50 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7effec03eda0 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.471+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efffc10d7e0 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.472+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7effec030020 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.472+0000 7efff3fff640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 0x7effe8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.472+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7effec066c40 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.472+0000 7efff27fc640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 0x7effe8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.472+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efffc102480 con 0x7efffc0fbf10 2024-06-06T03:10:28.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.473+0000 7efff27fc640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 0x7effe8036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7effe4006fd0 tx=0x7effe4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:28.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.477+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7effec033420 con 0x7efffc0fbf10 2024-06-06T03:10:28.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:28.655+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7efffc101e60 con 0x7effe8034900 2024-06-06T03:10:29.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:29 smithi153 ceph-mon[29300]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:31.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:31 smithi153 ceph-mon[29300]: Deploying cephadm binary to smithi173 2024-06-06T03:10:31.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:31 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:31.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:31 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.533+0000 7efff3fff640 1 -- 172.21.15.153:0/1883925158 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7effe4009510 con 0x7effe8034900 2024-06-06T03:10:31.534 INFO:teuthology.orchestra.run.smithi153.stdout:Added host 'smithi173' with addr '172.21.15.173' 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 msgr2=0x7effe8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 0x7effe8036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7effe4006fd0 tx=0x7effe4006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 msgr2=0x7efffc10c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7effec004820 tx=0x7effec009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 shutdown_connections 2024-06-06T03:10:31.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7effe8034900 0x7effe8036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:31.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 --2- 172.21.15.153:0/1883925158 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efffc0fbf10 0x7efffc10c510 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:31.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.535+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 >> 172.21.15.153:0/1883925158 conn(0x7efffc0f76f0 msgr2=0x7efffc1030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:31.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.536+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 shutdown_connections 2024-06-06T03:10:31.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:31.536+0000 7f000196e640 1 -- 172.21.15.153:0/1883925158 wait complete. 2024-06-06T03:10:32.189 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-06-06T03:10:32.443 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: Deploying daemon alertmanager.smithi153 on smithi153 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: Added host smithi173 2024-06-06T03:10:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:32 smithi153 ceph-mon[29300]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:33.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:33 smithi153 ceph-mon[29300]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T03:10:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:34 smithi153 ceph-mon[29300]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 -- 172.21.15.153:0/2881283037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 msgr2=0x7f11e40fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 --2- 172.21.15.153:0/2881283037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e40fb280 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f11e0004850 tx=0x7f11e0009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 -- 172.21.15.153:0/2881283037 shutdown_connections 2024-06-06T03:10:34.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 --2- 172.21.15.153:0/2881283037 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e40fb280 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 -- 172.21.15.153:0/2881283037 >> 172.21.15.153:0/2881283037 conn(0x7f11e40f65e0 msgr2=0x7f11e40f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 -- 172.21.15.153:0/2881283037 shutdown_connections 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.502+0000 7f11ec46e640 1 -- 172.21.15.153:0/2881283037 wait complete. 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ec46e640 1 Processor -- start 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ec46e640 1 -- start start 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ec46e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ec46e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e410ca80 con 0x7f11e40fae70 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ea20a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ea20a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49900/0 (socket says 172.21.15.153:49900) 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ea20a640 1 -- 172.21.15.153:0/372350074 learned_addr learned my addr 172.21.15.153:0/372350074 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ea20a640 1 -- 172.21.15.153:0/372350074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0005040 con 0x7f11e40fae70 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ea20a640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11e0004820 tx=0x7f11e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:34.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e0032650 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11e0032c80 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e410cc80 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.503+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f11e0033d60 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.504+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e410d810 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.504+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f11e0030020 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.504+0000 7f11daffd640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 0x7f11d0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.505+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f11e00666f0 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.505+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e4101390 con 0x7f11e40fae70 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.505+0000 7f11e9a09640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 0x7f11d0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.505+0000 7f11e9a09640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 0x7f11d0036e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006fd0 tx=0x7f11d4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:34.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.509+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f11e003c380 con 0x7f11e40fae70 2024-06-06T03:10:34.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.718+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f11e4065990 con 0x7f11d0034950 2024-06-06T03:10:34.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.719+0000 7f11daffd640 1 -- 172.21.15.153:0/372350074 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f11e4065990 con 0x7f11d0034950 2024-06-06T03:10:34.720 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:10:34.720 INFO:teuthology.orchestra.run.smithi153.stdout:[{"addr": "172.21.15.153", "hostname": "smithi153", "labels": [], "status": ""}, {"addr": "172.21.15.173", "hostname": "smithi173", "labels": [], "status": ""}] 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 msgr2=0x7f11d0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 0x7f11d0036e00 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f11d4006fd0 tx=0x7f11d4006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 msgr2=0x7f11e410c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f11e0004820 tx=0x7f11e0009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 shutdown_connections 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f11d0034950 0x7f11d0036e00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 --2- 172.21.15.153:0/372350074 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f11e40fae70 0x7f11e410c540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 >> 172.21.15.153:0/372350074 conn(0x7f11e40f65e0 msgr2=0x7f11e4065280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 shutdown_connections 2024-06-06T03:10:34.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:34.722+0000 7f11ec46e640 1 -- 172.21.15.153:0/372350074 wait complete. 2024-06-06T03:10:36.188 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-06T03:10:36.188 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-06-06T03:10:36.481 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:10:36.604 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:36 smithi153 ceph-mon[29300]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T03:10:36.604 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:36 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:36.604 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:36 smithi153 ceph-mon[29300]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:38.923 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:38 smithi153 ceph-mon[29300]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:40.909 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:40 smithi153 ceph-mon[29300]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 -- 172.21.15.153:0/1175538040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 msgr2=0x7f3b740fa790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 --2- 172.21.15.153:0/1175538040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b740fa790 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b64004850 tx=0x7f3b64009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 -- 172.21.15.153:0/1175538040 shutdown_connections 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 --2- 172.21.15.153:0/1175538040 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b740fa790 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 -- 172.21.15.153:0/1175538040 >> 172.21.15.153:0/1175538040 conn(0x7f3b740f5da0 msgr2=0x7f3b740f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 -- 172.21.15.153:0/1175538040 shutdown_connections 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.125+0000 7f3b79981640 1 -- 172.21.15.153:0/1175538040 wait complete. 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b79981640 1 Processor -- start 2024-06-06T03:10:41.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b79981640 1 -- start start 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b79981640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b79981640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b7410ca90 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b72ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b72ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51350/0 (socket says 172.21.15.153:51350) 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b72ffd640 1 -- 172.21.15.153:0/2003786775 learned_addr learned my addr 172.21.15.153:0/2003786775 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.126+0000 7f3b72ffd640 1 -- 172.21.15.153:0/2003786775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b64005040 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b72ffd640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b64004820 tx=0x7f3b64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b64032650 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3b64032c80 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b7410cc90 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b6403eda0 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.127+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b7410d820 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f3b64030020 con 0x7f3b740fa380 2024-06-06T03:10:41.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b6b7fe640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 0x7f3b60036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3b64066920 con 0x7f3b740fa380 2024-06-06T03:10:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b741024c0 con 0x7f3b740fa380 2024-06-06T03:10:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b6bfff640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 0x7f3b60036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.128+0000 7f3b6bfff640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 0x7f3b60036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006fd0 tx=0x7f3b5c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:41.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.132+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3b6403c510 con 0x7f3b740fa380 2024-06-06T03:10:41.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.308+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f3b7405ead0 con 0x7f3b740fa380 2024-06-06T03:10:41.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.628+0000 7f3b6b7fe640 1 -- 172.21.15.153:0/2003786775 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f3b6403c330 con 0x7f3b740fa380 2024-06-06T03:10:41.629 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 msgr2=0x7f3b60036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 0x7f3b60036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3b5c006fd0 tx=0x7f3b5c006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 msgr2=0x7f3b7410c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3b64004820 tx=0x7f3b64009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 shutdown_connections 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f3b60034900 0x7f3b60036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 --2- 172.21.15.153:0/2003786775 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b740fa380 0x7f3b7410c550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 >> 172.21.15.153:0/2003786775 conn(0x7f3b740f5da0 msgr2=0x7f3b741030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 shutdown_connections 2024-06-06T03:10:41.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:10:41.631+0000 7f3b79981640 1 -- 172.21.15.153:0/2003786775 wait complete. 2024-06-06T03:10:42.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:41 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2003786775' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-06T03:10:42.375 INFO:tasks.cephadm:Adding mon.smithi153 on smithi153 2024-06-06T03:10:42.375 INFO:tasks.cephadm:Adding mon.smithi173 on smithi173 2024-06-06T03:10:42.376 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch apply mon '2;smithi153:172.21.15.153=smithi153;smithi173:172.21.15.173=smithi173' 2024-06-06T03:10:42.623 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:42.671 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2003786775' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: osdmap e4: 0 total, 0 up, 0 in 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:43.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T03:10:43.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:42 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:44.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:43 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-06T03:10:44.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:43 smithi153 ceph-mon[29300]: Deploying daemon grafana.smithi153 on smithi153 2024-06-06T03:10:44.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- 172.21.15.173:0/2449357991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 msgr2=0x7f34200fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:44.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 --2- 172.21.15.173:0/2449357991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f34200fc360 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3418009bd0 tx=0x7f3418004d10 comp rx=0 tx=0).stop 2024-06-06T03:10:44.208 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- 172.21.15.173:0/2449357991 shutdown_connections 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 --2- 172.21.15.173:0/2449357991 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f34200fc360 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- 172.21.15.173:0/2449357991 >> 172.21.15.173:0/2449357991 conn(0x7f34200f7710 msgr2=0x7f34200f9b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- 172.21.15.173:0/2449357991 shutdown_connections 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- 172.21.15.173:0/2449357991 wait complete. 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 Processor -- start 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.207+0000 7f342585c640 1 -- start start 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f342585c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f342585c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f342010cab0 con 0x7f34200fbf50 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f341effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f341effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:39908/0 (socket says 172.21.15.173:39908) 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f341effd640 1 -- 172.21.15.173:0/1212019436 learned_addr learned my addr 172.21.15.173:0/1212019436 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f341effd640 1 -- 172.21.15.173:0/1212019436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34180098b0 con 0x7f34200fbf50 2024-06-06T03:10:44.209 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f341effd640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3418009ba0 tx=0x7f3418004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:44.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34180329b0 con 0x7f34200fbf50 2024-06-06T03:10:44.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f341803d420 con 0x7f34200fbf50 2024-06-06T03:10:44.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f342010ccb0 con 0x7f34200fbf50 2024-06-06T03:10:44.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.208+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3418031590 con 0x7f34200fbf50 2024-06-06T03:10:44.210 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.209+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f342010d100 con 0x7f34200fbf50 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.209+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f341802f030 con 0x7f34200fbf50 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.209+0000 7f34177fe640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 0x7f340c036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.209+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f342005ead0 con 0x7f34200fbf50 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.209+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f3418067db0 con 0x7f34200fbf50 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.210+0000 7f341e7fc640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 0x7f340c036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:44.211 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.210+0000 7f341e7fc640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 0x7f340c036d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:44.214 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.214+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3418041070 con 0x7f34200fbf50 2024-06-06T03:10:44.387 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.387+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 --> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}) v1 -- 0x7f3420105b50 con 0x7f340c0348b0 2024-06-06T03:10:44.393 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.393+0000 7f34177fe640 1 -- 172.21.15.173:0/1212019436 <== mgr.14160 v2:172.21.15.153:6800/659590756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3420105b50 con 0x7f340c0348b0 2024-06-06T03:10:44.393 INFO:teuthology.orchestra.run.smithi173.stdout:Scheduled mon update... 2024-06-06T03:10:44.394 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 msgr2=0x7f340c036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 0x7f340c036d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3408006fd0 tx=0x7f3408006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 msgr2=0x7f342010c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f3418009ba0 tx=0x7f3418004d10 comp rx=0 tx=0).stop 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 shutdown_connections 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f340c0348b0 0x7f340c036d60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 --2- 172.21.15.173:0/1212019436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f34200fbf50 0x7f342010c570 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 >> 172.21.15.173:0/1212019436 conn(0x7f34200f7710 msgr2=0x7f34201030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 shutdown_connections 2024-06-06T03:10:44.395 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:44.395+0000 7f342585c640 1 -- 172.21.15.173:0/1212019436 wait complete. 2024-06-06T03:10:44.966 DEBUG:teuthology.orchestra.run.smithi173:mon.smithi173> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi173.service 2024-06-06T03:10:44.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:10:44.969 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:10:45.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:44 smithi153 ceph-mon[29300]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:45.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:44 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:45.245 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:45.292 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:46.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:45 smithi153 ceph-mon[29300]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi153:172.21.15.153=smithi153;smithi173:172.21.15.173=smithi173", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:10:46.104 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:45 smithi153 ceph-mon[29300]: Saving service mon spec with placement smithi153:172.21.15.153=smithi153;smithi173:172.21.15.173=smithi173;count:2 2024-06-06T03:10:46.104 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:45 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 -- 172.21.15.173:0/1371582318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 msgr2=0x7f75e00fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 --2- 172.21.15.173:0/1371582318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e00fb450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f75d0004850 tx=0x7f75d0009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 -- 172.21.15.173:0/1371582318 shutdown_connections 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 --2- 172.21.15.173:0/1371582318 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e00fb450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 -- 172.21.15.173:0/1371582318 >> 172.21.15.173:0/1371582318 conn(0x7f75e00f6820 msgr2=0x7f75e00f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 -- 172.21.15.173:0/1371582318 shutdown_connections 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.624+0000 7f75e4942640 1 -- 172.21.15.173:0/1371582318 wait complete. 2024-06-06T03:10:46.625 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75e4942640 1 Processor -- start 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75e4942640 1 -- start start 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75e4942640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75e4942640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e010f460 con 0x7f75e00fb040 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75de59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75de59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:39926/0 (socket says 172.21.15.173:39926) 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75de59c640 1 -- 172.21.15.173:0/43300290 learned_addr learned my addr 172.21.15.173:0/43300290 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75de59c640 1 -- 172.21.15.173:0/43300290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d0005040 con 0x7f75e00fb040 2024-06-06T03:10:46.626 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.625+0000 7f75de59c640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75d0004820 tx=0x7f75d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:46.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.626+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d0032650 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.626+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f75d0032c80 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.626+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e010f5a0 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.626+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75d003eda0 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.626+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e010fa70 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f75d0030020 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e005ead0 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75d6ffd640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 0x7f75cc036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f75d0066990 con 0x7f75e00fb040 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75d7fff640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 0x7f75cc036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:46.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.627+0000 7f75d7fff640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 0x7f75cc036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75c8006fd0 tx=0x7f75c8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:46.635 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.635+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f75d00327b0 con 0x7f75e00fb040 2024-06-06T03:10:46.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.864+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75e004b480 con 0x7f75e00fb040 2024-06-06T03:10:46.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.864+0000 7f75d6ffd640 1 -- 172.21.15.173:0/43300290 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f75d003c330 con 0x7f75e00fb040 2024-06-06T03:10:46.864 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:10:46.864 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:10:46.864 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 msgr2=0x7f75cc036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 0x7f75cc036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f75c8006fd0 tx=0x7f75c8006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 msgr2=0x7f75e0065250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f75d0004820 tx=0x7f75d0009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 shutdown_connections 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f75cc034900 0x7f75cc036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 --2- 172.21.15.173:0/43300290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f75e00fb040 0x7f75e0065250 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 >> 172.21.15.173:0/43300290 conn(0x7f75e00f6820 msgr2=0x7f75e01021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:46.866 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 shutdown_connections 2024-06-06T03:10:46.867 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:46.866+0000 7f75e4942640 1 -- 172.21.15.173:0/43300290 wait complete. 2024-06-06T03:10:47.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:46 smithi153 ceph-mon[29300]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:48.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:47 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/43300290' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:10:48.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:10:48.402 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:10:48.650 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:48.697 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:49.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:48 smithi153 ceph-mon[29300]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.035+0000 7f2bff17a640 1 -- 172.21.15.173:0/439377876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 msgr2=0x7f2bf80fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.035+0000 7f2bff17a640 1 --2- 172.21.15.173:0/439377876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf80fb280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2be8004850 tx=0x7f2be8009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.035+0000 7f2bff17a640 1 -- 172.21.15.173:0/439377876 shutdown_connections 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 --2- 172.21.15.173:0/439377876 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf80fb280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 -- 172.21.15.173:0/439377876 >> 172.21.15.173:0/439377876 conn(0x7f2bf80f65e0 msgr2=0x7f2bf80f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 -- 172.21.15.173:0/439377876 shutdown_connections 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 -- 172.21.15.173:0/439377876 wait complete. 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 Processor -- start 2024-06-06T03:10:50.043 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 -- start start 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.036+0000 7f2bff17a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf810cab0 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bfcf16640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bfcf16640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:39948/0 (socket says 172.21.15.173:39948) 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bfcf16640 1 -- 172.21.15.173:0/1631875955 learned_addr learned my addr 172.21.15.173:0/1631875955 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bfcf16640 1 -- 172.21.15.173:0/1631875955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8005040 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bfcf16640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2be8004820 tx=0x7f2be8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be8032650 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf810ccb0 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2be8032c80 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2be8033d60 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.037+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf810d840 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.038+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f2be8030020 con 0x7f2bf80fae70 2024-06-06T03:10:50.044 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.038+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf8101390 con 0x7f2bf80fae70 2024-06-06T03:10:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.038+0000 7f2bf57fa640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 0x7f2be0036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.038+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f2be80667c0 con 0x7f2bf80fae70 2024-06-06T03:10:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.038+0000 7f2bf7fff640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 0x7f2be0036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.039+0000 7f2bf7fff640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 0x7f2be0036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2bec006fd0 tx=0x7f2bec006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:50.045 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.043+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2be803c560 con 0x7f2bf80fae70 2024-06-06T03:10:50.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.272+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2bf810da80 con 0x7f2bf80fae70 2024-06-06T03:10:50.272 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.272+0000 7f2bf57fa640 1 -- 172.21.15.173:0/1631875955 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f2be803c380 con 0x7f2bf80fae70 2024-06-06T03:10:50.272 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:10:50.272 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:10:50.273 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:10:50.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 msgr2=0x7f2be0036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:50.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 0x7f2be0036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2bec006fd0 tx=0x7f2bec006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.274 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 msgr2=0x7f2bf810c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2be8004820 tx=0x7f2be8009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 shutdown_connections 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f2be0034900 0x7f2be0036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 --2- 172.21.15.173:0/1631875955 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2bf80fae70 0x7f2bf810c570 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 >> 172.21.15.173:0/1631875955 conn(0x7f2bf80f65e0 msgr2=0x7f2bf8065260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 shutdown_connections 2024-06-06T03:10:50.275 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:50.274+0000 7f2bff17a640 1 -- 172.21.15.173:0/1631875955 wait complete. 2024-06-06T03:10:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:50 smithi153 ceph-mon[29300]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:50 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1631875955' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:10:51.841 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:10:51.841 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:10:52.077 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:52.124 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:52.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:52 smithi153 ceph-mon[29300]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:53.416 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 -- 172.21.15.173:0/3978111255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 msgr2=0x7f6be40fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 --2- 172.21.15.173:0/3978111255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be40fb280 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0048d0 tx=0x7f6bdc009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 -- 172.21.15.173:0/3978111255 shutdown_connections 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 --2- 172.21.15.173:0/3978111255 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be40fb280 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 -- 172.21.15.173:0/3978111255 >> 172.21.15.173:0/3978111255 conn(0x7f6be40f65e0 msgr2=0x7f6be40f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 -- 172.21.15.173:0/3978111255 shutdown_connections 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 -- 172.21.15.173:0/3978111255 wait complete. 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.415+0000 7f6bec254640 1 Processor -- start 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6bec254640 1 -- start start 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6bec254640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:53.417 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6bec254640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be410a950 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6be9ff0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6be9ff0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:50522/0 (socket says 172.21.15.173:50522) 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6be9ff0640 1 -- 172.21.15.173:0/3209841924 learned_addr learned my addr 172.21.15.173:0/3209841924 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6be9ff0640 1 -- 172.21.15.173:0/3209841924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bdc005040 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.416+0000 7f6be9ff0640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0048a0 tx=0x7f6bdc038040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.417+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bdc034ba0 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.417+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bdc034d00 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.417+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be410ab50 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.417+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bdc040ab0 con 0x7f6be40fae70 2024-06-06T03:10:53.418 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.417+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be410afa0 con 0x7f6be40fae70 2024-06-06T03:10:53.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.418+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f6bdc009070 con 0x7f6be40fae70 2024-06-06T03:10:53.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.418+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be405ead0 con 0x7f6be40fae70 2024-06-06T03:10:53.419 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.418+0000 7f6be27fc640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 0x7f6bcc036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:53.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.418+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6bdc068c20 con 0x7f6be40fae70 2024-06-06T03:10:53.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.418+0000 7f6be97ef640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 0x7f6bcc036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:53.420 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.419+0000 7f6be97ef640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 0x7f6bcc036e00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:53.422 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.423+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6bdc03e9d0 con 0x7f6be40fae70 2024-06-06T03:10:53.651 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.651+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6be404b480 con 0x7f6be40fae70 2024-06-06T03:10:53.652 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.652+0000 7f6be27fc640 1 -- 172.21.15.173:0/3209841924 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6bdc045020 con 0x7f6be40fae70 2024-06-06T03:10:53.652 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:10:53.652 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:10:53.652 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:10:53.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 msgr2=0x7f6bcc036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:53.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 0x7f6bcc036e00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6bd8006fd0 tx=0x7f6bd8006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:53.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 msgr2=0x7f6be410a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:53.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f6bdc0048a0 tx=0x7f6bdc038040 comp rx=0 tx=0).stop 2024-06-06T03:10:53.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 shutdown_connections 2024-06-06T03:10:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f6bcc034950 0x7f6bcc036e00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 --2- 172.21.15.173:0/3209841924 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6be40fae70 0x7f6be410a410 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 >> 172.21.15.173:0/3209841924 conn(0x7f6be40f65e0 msgr2=0x7f6be4100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 shutdown_connections 2024-06-06T03:10:53.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:53.654+0000 7f6bec254640 1 -- 172.21.15.173:0/3209841924 wait complete. 2024-06-06T03:10:54.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:54 smithi153 ceph-mon[29300]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:54.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:54 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3209841924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:10:55.299 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:10:55.300 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:10:55.549 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:55.595 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: Deploying daemon prometheus.smithi153 on smithi153 2024-06-06T03:10:56.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:56 smithi153 ceph-mon[29300]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:57.002 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 -- 172.21.15.173:0/3647826855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 msgr2=0x7f177c0fb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 --2- 172.21.15.173:0/3647826855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c0fb300 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f176c004850 tx=0x7f176c009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 -- 172.21.15.173:0/3647826855 shutdown_connections 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 --2- 172.21.15.173:0/3647826855 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c0fb300 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 -- 172.21.15.173:0/3647826855 >> 172.21.15.173:0/3647826855 conn(0x7f177c0f6610 msgr2=0x7f177c0f8a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 -- 172.21.15.173:0/3647826855 shutdown_connections 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.000+0000 7f178231b640 1 -- 172.21.15.173:0/3647826855 wait complete. 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f178231b640 1 Processor -- start 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f178231b640 1 -- start start 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f178231b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f178231b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c10cad0 con 0x7f177c0faef0 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f177bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:57.003 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f177bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:50536/0 (socket says 172.21.15.173:50536) 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f177bfff640 1 -- 172.21.15.173:0/3806009344 learned_addr learned my addr 172.21.15.173:0/3806009344 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f177bfff640 1 -- 172.21.15.173:0/3806009344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c005040 con 0x7f177c0faef0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.001+0000 7f177bfff640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f176c004820 tx=0x7f176c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.002+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176c032650 con 0x7f177c0faef0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.002+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f176c032c80 con 0x7f177c0faef0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.002+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f176c03eda0 con 0x7f177c0faef0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.002+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c10ccd0 con 0x7f177c0faef0 2024-06-06T03:10:57.004 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.002+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c10d860 con 0x7f177c0faef0 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.003+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f176c030020 con 0x7f177c0faef0 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.003+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c1013c0 con 0x7f177c0faef0 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.003+0000 7f17797fa640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 0x7f1768036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.003+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f176c066940 con 0x7f177c0faef0 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.003+0000 7f1773fff640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 0x7f1768036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:10:57.005 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.004+0000 7f1773fff640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 0x7f1768036e00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1764006fd0 tx=0x7f1764006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:10:57.009 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.008+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f176c0327b0 con 0x7f177c0faef0 2024-06-06T03:10:57.234 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.234+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f177c10daa0 con 0x7f177c0faef0 2024-06-06T03:10:57.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.234+0000 7f17797fa640 1 -- 172.21.15.173:0/3806009344 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f176c03c330 con 0x7f177c0faef0 2024-06-06T03:10:57.235 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:10:57.235 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:10:57.236 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 msgr2=0x7f1768036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 0x7f1768036e00 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1764006fd0 tx=0x7f1764006db0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 msgr2=0x7f177c10c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f176c004820 tx=0x7f176c009da0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 shutdown_connections 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f1768034950 0x7f1768036e00 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 --2- 172.21.15.173:0/3806009344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f177c0faef0 0x7f177c10c590 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 >> 172.21.15.173:0/3806009344 conn(0x7f177c0f6610 msgr2=0x7f177c0651c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 shutdown_connections 2024-06-06T03:10:57.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:10:57.236+0000 7f178231b640 1 -- 172.21.15.173:0/3806009344 wait complete. 2024-06-06T03:10:57.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:57 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3806009344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:10:58.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:10:58 smithi153 ceph-mon[29300]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:10:58.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:10:58.811 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:10:59.060 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:10:59.107 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:00.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- 172.21.15.173:0/1051303815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 msgr2=0x7faa500fc920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:00.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 --2- 172.21.15.173:0/1051303815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa500fc920 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004850 tx=0x7faa4c009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- 172.21.15.173:0/1051303815 shutdown_connections 2024-06-06T03:11:00.654 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 --2- 172.21.15.173:0/1051303815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa500fc920 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- 172.21.15.173:0/1051303815 >> 172.21.15.173:0/1051303815 conn(0x7faa500f65e0 msgr2=0x7faa500f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- 172.21.15.173:0/1051303815 shutdown_connections 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- 172.21.15.173:0/1051303815 wait complete. 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 Processor -- start 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- start start 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa576f8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa501130e0 con 0x7faa50065f20 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa55494640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.653+0000 7faa55494640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:50554/0 (socket says 172.21.15.173:50554) 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa55494640 1 -- 172.21.15.173:0/2843935524 learned_addr learned my addr 172.21.15.173:0/2843935524 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:00.655 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa55494640 1 -- 172.21.15.173:0/2843935524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c005040 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa55494640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004820 tx=0x7faa4c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa4c032650 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa501132e0 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faa4c032c80 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faa4c033d60 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.654+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa50113e70 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.655+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7faa4c030020 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.655+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa50100280 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.655+0000 7faa45ffb640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 0x7faa3c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.655+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7faa4c0667c0 con 0x7faa50065f20 2024-06-06T03:11:00.657 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.656+0000 7faa54c93640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 0x7faa3c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:00.658 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.656+0000 7faa54c93640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 0x7faa3c036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:00.661 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.662+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faa4c03c560 con 0x7faa50065f20 2024-06-06T03:11:00.887 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.887+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faa50114870 con 0x7faa50065f20 2024-06-06T03:11:00.889 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.888+0000 7faa45ffb640 1 -- 172.21.15.173:0/2843935524 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7faa4c03c380 con 0x7faa50065f20 2024-06-06T03:11:00.889 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:00.889 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:00.889 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 msgr2=0x7faa3c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 0x7faa3c036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7faa40006fd0 tx=0x7faa40006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 msgr2=0x7faa50112ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7faa4c004820 tx=0x7faa4c009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 shutdown_connections 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7faa3c034900 0x7faa3c036db0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 --2- 172.21.15.173:0/2843935524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faa50065f20 0x7faa50112ba0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 >> 172.21.15.173:0/2843935524 conn(0x7faa500f65e0 msgr2=0x7faa50109740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 shutdown_connections 2024-06-06T03:11:00.891 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:00.891+0000 7faa576f8640 1 -- 172.21.15.173:0/2843935524 wait complete. 2024-06-06T03:11:01.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:00 smithi153 ceph-mon[29300]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:01.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:00 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:01.995 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/2843935524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:02.472 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:02.472 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:02.713 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:02.761 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:02.979 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:02 smithi153 ceph-mon[29300]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 -- 172.21.15.173:0/401567133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 msgr2=0x7fbf340fb280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 --2- 172.21.15.173:0/401567133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf340fb280 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbf24004850 tx=0x7fbf24009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 -- 172.21.15.173:0/401567133 shutdown_connections 2024-06-06T03:11:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 --2- 172.21.15.173:0/401567133 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf340fb280 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.018 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 -- 172.21.15.173:0/401567133 >> 172.21.15.173:0/401567133 conn(0x7fbf340f65e0 msgr2=0x7fbf340f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 -- 172.21.15.173:0/401567133 shutdown_connections 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.016+0000 7fbf3a864640 1 -- 172.21.15.173:0/401567133 wait complete. 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf3a864640 1 Processor -- start 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf3a864640 1 -- start start 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf3a864640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf3a864640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf3410caa0 con 0x7fbf340fae70 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf33fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:55270/0 (socket says 172.21.15.173:55270) 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.017+0000 7fbf33fff640 1 -- 172.21.15.173:0/2866732306 learned_addr learned my addr 172.21.15.173:0/2866732306 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf33fff640 1 -- 172.21.15.173:0/2866732306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24005040 con 0x7fbf340fae70 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf33fff640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf240098d0 tx=0x7fbf24009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf24032650 con 0x7fbf340fae70 2024-06-06T03:11:04.019 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf3410cca0 con 0x7fbf340fae70 2024-06-06T03:11:04.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf24032c80 con 0x7fbf340fae70 2024-06-06T03:11:04.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf2403dc50 con 0x7fbf340fae70 2024-06-06T03:11:04.020 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.018+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf3410d110 con 0x7fbf340fae70 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.019+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fbf24030020 con 0x7fbf340fae70 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.019+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf3405ead0 con 0x7fbf340fae70 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.019+0000 7fbf30ff9640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 0x7fbf1c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.019+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fbf24066d60 con 0x7fbf340fae70 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.020+0000 7fbf337fe640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 0x7fbf1c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:04.021 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.020+0000 7fbf337fe640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 0x7fbf1c036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbf28006fd0 tx=0x7fbf28006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:04.024 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.024+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbf240327b0 con 0x7fbf340fae70 2024-06-06T03:11:04.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.249+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf3404b480 con 0x7fbf340fae70 2024-06-06T03:11:04.249 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.250+0000 7fbf30ff9640 1 -- 172.21.15.173:0/2866732306 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fbf24045390 con 0x7fbf340fae70 2024-06-06T03:11:04.250 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:04.250 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:04.250 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 msgr2=0x7fbf1c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 0x7fbf1c036db0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbf28006fd0 tx=0x7fbf28006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 msgr2=0x7fbf3410c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf240098d0 tx=0x7fbf24009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 shutdown_connections 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7fbf1c034900 0x7fbf1c036db0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 --2- 172.21.15.173:0/2866732306 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf340fae70 0x7fbf3410c560 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 >> 172.21.15.173:0/2866732306 conn(0x7fbf340f65e0 msgr2=0x7fbf34065280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 shutdown_connections 2024-06-06T03:11:04.252 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:04.252+0000 7fbf3a864640 1 -- 172.21.15.173:0/2866732306 wait complete. 2024-06-06T03:11:04.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:04 smithi153 ceph-mon[29300]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:04.931 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:04 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/2866732306' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:05.777 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:05.777 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:06.027 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:06.074 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:06.546 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:06 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:06.546 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:06 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:06.546 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:06 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:06.547 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:06 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-06T03:11:06.547 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:06 smithi153 ceph-mon[29300]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:07.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:07 smithi153 ceph-mon[29300]: from='mgr.14160 172.21.15.153:0/605731221' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-06T03:11:07.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:07 smithi153 ceph-mon[29300]: mgrmap e14: smithi153.gidtue(active, since 56s) 2024-06-06T03:11:07.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.562+0000 7f909960c640 1 -- 172.21.15.173:0/1938872990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 msgr2=0x7f90940fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:07.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.562+0000 7f909960c640 1 --2- 172.21.15.173:0/1938872990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f90940fc320 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9084004850 tx=0x7f9084009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- 172.21.15.173:0/1938872990 shutdown_connections 2024-06-06T03:11:07.563 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 --2- 172.21.15.173:0/1938872990 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f90940fc320 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- 172.21.15.173:0/1938872990 >> 172.21.15.173:0/1938872990 conn(0x7f90940f76f0 msgr2=0x7f90940f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- 172.21.15.173:0/1938872990 shutdown_connections 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- 172.21.15.173:0/1938872990 wait complete. 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 Processor -- start 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- start start 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.563+0000 7f909960c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909410ca50 con 0x7f90940fbf10 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f9092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f9092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:55300/0 (socket says 172.21.15.173:55300) 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f9092ffd640 1 -- 172.21.15.173:0/3850409823 learned_addr learned my addr 172.21.15.173:0/3850409823 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:07.564 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f9092ffd640 1 -- 172.21.15.173:0/3850409823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9084005040 con 0x7f90940fbf10 2024-06-06T03:11:07.565 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f9092ffd640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9084004820 tx=0x7f9084009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9084032650 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9084032c80 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909410cc50 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908403eda0 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.564+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909410d7e0 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 14) v1 ==== 42017+0+0 (secure 0 0 0) 0x7f9084030020 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f907b7fe640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 0x7f907c036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9084066c70 con 0x7f90940fbf10 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f90927fc640 1 -- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 msgr2=0x7f907c036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/659590756 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f90927fc640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 0x7f907c036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T03:11:07.566 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.565+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9094102480 con 0x7f90940fbf10 2024-06-06T03:11:07.573 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.574+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9084033420 con 0x7f90940fbf10 2024-06-06T03:11:07.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.766+0000 7f90927fc640 1 -- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 msgr2=0x7f907c036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:6800/659590756 2024-06-06T03:11:07.765 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.766+0000 7f90927fc640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 0x7f907c036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-06T03:11:07.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.799+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f909405ead0 con 0x7f90940fbf10 2024-06-06T03:11:07.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.800+0000 7f907b7fe640 1 -- 172.21.15.173:0/3850409823 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f909405ead0 con 0x7f90940fbf10 2024-06-06T03:11:07.799 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:07.800 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:07.800 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:07.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.801+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 msgr2=0x7f907c036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T03:11:07.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.801+0000 7f909960c640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 0x7f907c036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.801+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 msgr2=0x7f909410c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:07.801 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.801+0000 7f909960c640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9084004820 tx=0x7f9084009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 shutdown_connections 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:6800/659590756,v1:172.21.15.153:6801/659590756] conn(0x7f907c034950 0x7f907c036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 --2- 172.21.15.173:0/3850409823 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f90940fbf10 0x7f909410c510 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 >> 172.21.15.173:0/3850409823 conn(0x7f90940f76f0 msgr2=0x7f90941030b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 shutdown_connections 2024-06-06T03:11:07.802 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:07.802+0000 7f909960c640 1 -- 172.21.15.173:0/3850409823 wait complete. 2024-06-06T03:11:08.436 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:08 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3850409823' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:09.315 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:09.315 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:09.563 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:09.610 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:10.874 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: Active manager daemon smithi153.gidtue restarted 2024-06-06T03:11:10.874 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: Activating manager daemon smithi153.gidtue 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: mgrmap e15: smithi153.gidtue(active, starting, since 0.00302462s) 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr metadata", "who": "smithi153.gidtue", "id": "smithi153.gidtue"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: Manager daemon smithi153.gidtue is now available 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:10.875 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 -- 172.21.15.173:0/2470995156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 msgr2=0x7f8bac0fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 --2- 172.21.15.173:0/2470995156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac0fb450 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4004850 tx=0x7f8ba4009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 -- 172.21.15.173:0/2470995156 shutdown_connections 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 --2- 172.21.15.173:0/2470995156 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac0fb450 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 -- 172.21.15.173:0/2470995156 >> 172.21.15.173:0/2470995156 conn(0x7f8bac0f6820 msgr2=0x7f8bac0f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 -- 172.21.15.173:0/2470995156 shutdown_connections 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 -- 172.21.15.173:0/2470995156 wait complete. 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.054+0000 7f8bb3e04640 1 Processor -- start 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb3e04640 1 -- start start 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb3e04640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb3e04640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bac10bb80 con 0x7f8bac0fb040 2024-06-06T03:11:11.072 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb1ba0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb1ba0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:55318/0 (socket says 172.21.15.173:55318) 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb1ba0640 1 -- 172.21.15.173:0/3083127619 learned_addr learned my addr 172.21.15.173:0/3083127619 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb1ba0640 1 -- 172.21.15.173:0/3083127619 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba4005040 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb1ba0640 1 --2- 172.21.15.173:0/3083127619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4004820 tx=0x7f8ba4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba4032650 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac10bd80 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ba4032c80 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.055+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba403eda0 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.056+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac10c910 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.056+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 15) v1 ==== 41740+0+0 (secure 0 0 0) 0x7f8ba4030020 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.056+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bac1015b0 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.056+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f8ba4066660 con 0x7f8bac0fb040 2024-06-06T03:11:11.073 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.065+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8ba403c510 con 0x7f8bac0fb040 2024-06-06T03:11:11.303 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.303+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bac0008d0 con 0x7f8bac0fb040 2024-06-06T03:11:11.304 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.304+0000 7f8baa7fc640 1 -- 172.21.15.173:0/3083127619 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8ba403c330 con 0x7f8bac0fb040 2024-06-06T03:11:11.304 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:11.304 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:11.304 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 msgr2=0x7f8bac10b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 --2- 172.21.15.173:0/3083127619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f8ba4004820 tx=0x7f8ba4009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 shutdown_connections 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 --2- 172.21.15.173:0/3083127619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8bac0fb040 0x7f8bac10b640 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 >> 172.21.15.173:0/3083127619 conn(0x7f8bac0f6820 msgr2=0x7f8bac1021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 shutdown_connections 2024-06-06T03:11:11.307 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:11.307+0000 7f8bb3e04640 1 -- 172.21.15.173:0/3083127619 wait complete. 2024-06-06T03:11:11.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/mirror_snapshot_schedule"}]: dispatch 2024-06-06T03:11:11.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi153.gidtue/trash_purge_schedule"}]: dispatch 2024-06-06T03:11:11.944 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:11 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3083127619' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:11.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:11.945 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:11 smithi153 ceph-mon[29300]: mgrmap e16: smithi153.gidtue(active, since 1.00578s) 2024-06-06T03:11:12.861 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:12.861 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:11:11] ENGINE Bus STARTING 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:11:11] ENGINE Serving on https://172.21.15.153:7150 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:11:11] ENGINE Bus STARTED 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: [06/Jun/2024:03:11:11] ENGINE Error in HTTPServer.serve 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: Traceback (most recent call last): 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._connections.run(self.expiration_interval) 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._run(expiration_interval) 2024-06-06T03:11:13.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: s = self.context.wrap_socket( 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: return self.sslsocket_class._create( 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self.do_handshake() 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: self._sslobj.do_handshake() 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-06T03:11:13.226 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:13.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:13.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:12 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:13.921 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:14.410 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-06T03:11:14.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:14 smithi153 ceph-mon[29300]: mgrmap e17: smithi153.gidtue(active, since 2s) 2024-06-06T03:11:16.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd/host:smithi153", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd/host:smithi173", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:16.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:11:16.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 -- 172.21.15.173:0/2104397134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 msgr2=0x7f69a8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:16.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 --2- 172.21.15.173:0/2104397134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a8102310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6998004850 tx=0x7f6998009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 -- 172.21.15.173:0/2104397134 shutdown_connections 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 --2- 172.21.15.173:0/2104397134 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a8102310 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 -- 172.21.15.173:0/2104397134 >> 172.21.15.173:0/2104397134 conn(0x7f69a8070e10 msgr2=0x7f69a8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 -- 172.21.15.173:0/2104397134 shutdown_connections 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.459+0000 7f69af915640 1 -- 172.21.15.173:0/2104397134 wait complete. 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69af915640 1 Processor -- start 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69af915640 1 -- start start 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69af915640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69af915640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a8112af0 con 0x7f69a8101f40 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69ad6b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:16.461 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69ad6b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:48950/0 (socket says 172.21.15.173:48950) 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69ad6b1640 1 -- 172.21.15.173:0/2042608867 learned_addr learned my addr 172.21.15.173:0/2042608867 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.460+0000 7f69ad6b1640 1 -- 172.21.15.173:0/2042608867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6998005040 con 0x7f69a8101f40 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f69ad6b1640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6998004820 tx=0x7f6998009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f699802cf10 con 0x7f69a8101f40 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6998033b00 con 0x7f69a8101f40 2024-06-06T03:11:16.462 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a8112cf0 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f699803db70 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.461+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a8113160 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f6998030020 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f699dffb640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 0x7f6994036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f699806a7f0 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f69aceb0640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 0x7f6994036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a80fd6b0 con 0x7f69a8101f40 2024-06-06T03:11:16.463 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.462+0000 7f69aceb0640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 0x7f6994036ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f69a4006fd0 tx=0x7f69a4006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:16.466 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.467+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6998032e40 con 0x7f69a8101f40 2024-06-06T03:11:16.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.720+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69a81062f0 con 0x7f69a8101f40 2024-06-06T03:11:16.720 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.721+0000 7f699dffb640 1 -- 172.21.15.173:0/2042608867 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f699803a2d0 con 0x7f69a8101f40 2024-06-06T03:11:16.720 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:16.721 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:16.721 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:16.722 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 msgr2=0x7f6994036ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 0x7f6994036ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f69a4006fd0 tx=0x7f69a4006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 msgr2=0x7f69a81125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6998004820 tx=0x7f6998009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 shutdown_connections 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6994034a40 0x7f6994036ef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 --2- 172.21.15.173:0/2042608867 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69a8101f40 0x7f69a81125b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 >> 172.21.15.173:0/2042608867 conn(0x7f69a8070e10 msgr2=0x7f69a8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 shutdown_connections 2024-06-06T03:11:16.723 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:16.723+0000 7f69af915640 1 -- 172.21.15.173:0/2042608867 wait complete. 2024-06-06T03:11:17.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi153:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi173:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi153:/etc/ceph/ceph.client.admin.keyring 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi173:/etc/ceph/ceph.client.admin.keyring 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi153:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: Updating smithi173:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:17.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:17.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:17.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:11:17.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-06T03:11:17.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:17.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:16 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/2042608867' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:18.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:17 smithi153 ceph-mon[29300]: Deploying daemon crash.smithi173 on smithi173 2024-06-06T03:11:19.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:19.478 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:19.733 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:20 smithi153 ceph-mon[29300]: Deploying daemon node-exporter.smithi173 on smithi173 2024-06-06T03:11:21.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.405+0000 7f958cb83640 1 -- 172.21.15.173:0/1175768280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 msgr2=0x7f95881011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:21.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.405+0000 7f958cb83640 1 --2- 172.21.15.173:0/1175768280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f95881011a0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f9578004850 tx=0x7f9578009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.457 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- 172.21.15.173:0/1175768280 shutdown_connections 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 --2- 172.21.15.173:0/1175768280 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f95881011a0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- 172.21.15.173:0/1175768280 >> 172.21.15.173:0/1175768280 conn(0x7f9588070da0 msgr2=0x7f95880711a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- 172.21.15.173:0/1175768280 shutdown_connections 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- 172.21.15.173:0/1175768280 wait complete. 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 Processor -- start 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- start start 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958cb83640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958810da90 con 0x7f9588100dd0 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:48990/0 (socket says 172.21.15.173:48990) 2024-06-06T03:11:21.458 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.406+0000 7f958659c640 1 -- 172.21.15.173:0/1521753460 learned_addr learned my addr 172.21.15.173:0/1521753460 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f958659c640 1 -- 172.21.15.173:0/1521753460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9578005040 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f958659c640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9578004820 tx=0x7f9578009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957802cf10 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958810dc90 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9578033b00 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957803db70 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.407+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958810e100 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.408+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f9578030020 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.408+0000 7f957effd640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 0x7f9574036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.408+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588103490 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.408+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f9578067000 con 0x7f9588100dd0 2024-06-06T03:11:21.459 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.409+0000 7f957ffff640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 0x7f9574036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:21.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.409+0000 7f957ffff640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 0x7f9574036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9570006fd0 tx=0x7f9570006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:21.460 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.413+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9578045570 con 0x7f9588100dd0 2024-06-06T03:11:21.673 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.674+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95880fc540 con 0x7f9588100dd0 2024-06-06T03:11:21.674 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.674+0000 7f957effd640 1 -- 172.21.15.173:0/1521753460 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9578045390 con 0x7f9588100dd0 2024-06-06T03:11:21.674 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:21.674 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:21.675 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 msgr2=0x7f9574036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 0x7f9574036ea0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9570006fd0 tx=0x7f9570006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 msgr2=0x7f958810d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9578004820 tx=0x7f9578009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 shutdown_connections 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f95740349f0 0x7f9574036ea0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 --2- 172.21.15.173:0/1521753460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9588100dd0 0x7f958810d550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 >> 172.21.15.173:0/1521753460 conn(0x7f9588070da0 msgr2=0x7f95880f9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:21.676 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 shutdown_connections 2024-06-06T03:11:21.677 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:21.676+0000 7f958cb83640 1 -- 172.21.15.173:0/1521753460 wait complete. 2024-06-06T03:11:22.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:21 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:22.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:21 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1521753460' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:23.901 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:23.901 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.zwtggy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T03:11:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.zwtggy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-06T03:11:24.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T03:11:24.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:24.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:24 smithi153 ceph-mon[29300]: Deploying daemon mgr.smithi173.zwtggy on smithi173 2024-06-06T03:11:24.825 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:26.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- 172.21.15.173:0/3232394563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 msgr2=0x7fd488101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:26.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/3232394563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488101210 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd478004850 tx=0x7fd478009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.606 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- 172.21.15.173:0/3232394563 shutdown_connections 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/3232394563 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488101210 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- 172.21.15.173:0/3232394563 >> 172.21.15.173:0/3232394563 conn(0x7fd488070e10 msgr2=0x7fd488071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- 172.21.15.173:0/3232394563 shutdown_connections 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- 172.21.15.173:0/3232394563 wait complete. 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 Processor -- start 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.606+0000 7fd48d9f2640 1 -- start start 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd48d9f2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd48d9f2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488112b60 con 0x7fd488100e40 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd486ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd486ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:47044/0 (socket says 172.21.15.173:47044) 2024-06-06T03:11:26.607 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd486ffd640 1 -- 172.21.15.173:0/1462512414 learned_addr learned my addr 172.21.15.173:0/1462512414 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd486ffd640 1 -- 172.21.15.173:0/1462512414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd478005040 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.607+0000 7fd486ffd640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd478004820 tx=0x7fd478009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.608+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd47802cf10 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.608+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488112d60 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.608+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd478033b00 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.608+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488113150 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.608+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd47803db70 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.609+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7fd478030020 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.609+0000 7fd46f7fe640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 0x7fd470036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.609+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd4780670b0 con 0x7fd488100e40 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.609+0000 7fd4867fc640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 0x7fd470036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:26.609 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.609+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4881122d0 con 0x7fd488100e40 2024-06-06T03:11:26.610 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.610+0000 7fd4867fc640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 0x7fd470036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:26.614 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.614+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd47803b780 con 0x7fd488100e40 2024-06-06T03:11:26.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.861+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd48804b480 con 0x7fd488100e40 2024-06-06T03:11:26.861 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.861+0000 7fd46f7fe640 1 -- 172.21.15.173:0/1462512414 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd478044300 con 0x7fd488100e40 2024-06-06T03:11:26.862 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:26.862 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:26.862 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:26.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 msgr2=0x7fd470036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:26.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 0x7fd470036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd47c006fd0 tx=0x7fd47c006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 msgr2=0x7fd488112620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:26.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd478004820 tx=0x7fd478009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 shutdown_connections 2024-06-06T03:11:26.864 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd4700349f0 0x7fd470036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 --2- 172.21.15.173:0/1462512414 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd488100e40 0x7fd488112620 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:26.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 >> 172.21.15.173:0/1462512414 conn(0x7fd488070e10 msgr2=0x7fd488064730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:26.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 shutdown_connections 2024-06-06T03:11:26.865 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:26.864+0000 7fd48d9f2640 1 -- 172.21.15.173:0/1462512414 wait complete. 2024-06-06T03:11:27.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:27.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:28.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1462512414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:28.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:28.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T03:11:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:28.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:27 smithi153 ceph-mon[29300]: Deploying daemon mon.smithi173 on smithi173 2024-06-06T03:11:28.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:28.703 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:29.456 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- 172.21.15.173:0/1029210550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 msgr2=0x7f54fc101210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 --2- 172.21.15.173:0/1029210550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc101210 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f54f8004850 tx=0x7f54f8009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- 172.21.15.173:0/1029210550 shutdown_connections 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 --2- 172.21.15.173:0/1029210550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc101210 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- 172.21.15.173:0/1029210550 >> 172.21.15.173:0/1029210550 conn(0x7f54fc070e10 msgr2=0x7f54fc071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- 172.21.15.173:0/1029210550 shutdown_connections 2024-06-06T03:11:31.342 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- 172.21.15.173:0/1029210550 wait complete. 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 Processor -- start 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- start start 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.342+0000 7f5503472640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc0656a0 con 0x7f54fc100e40 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f550120e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f550120e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:47082/0 (socket says 172.21.15.173:47082) 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f550120e640 1 -- 172.21.15.173:0/3186161443 learned_addr learned my addr 172.21.15.173:0/3186161443 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f550120e640 1 -- 172.21.15.173:0/3186161443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f8005040 con 0x7f54fc100e40 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f550120e640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54f8004820 tx=0x7f54f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f802cf10 con 0x7f54fc100e40 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54f8033b00 con 0x7f54fc100e40 2024-06-06T03:11:31.344 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc10ba20 con 0x7f54fc100e40 2024-06-06T03:11:31.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.343+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f803db70 con 0x7f54fc100e40 2024-06-06T03:11:31.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.344+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc10bef0 con 0x7f54fc100e40 2024-06-06T03:11:31.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.344+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f54f8030020 con 0x7f54fc100e40 2024-06-06T03:11:31.345 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.345+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc10c130 con 0x7f54fc100e40 2024-06-06T03:11:31.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.345+0000 7f54f1ffb640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 0x7f54e8036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:31.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.345+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f54f8067090 con 0x7f54fc100e40 2024-06-06T03:11:31.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.345+0000 7f5500a0d640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 0x7f54e8036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:31.346 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.345+0000 7f5500a0d640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 0x7f54e8036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec006db0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:31.349 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.350+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f54f803b780 con 0x7f54fc100e40 2024-06-06T03:11:31.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.596+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54fc04b480 con 0x7f54fc100e40 2024-06-06T03:11:31.596 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.597+0000 7f54f1ffb640 1 -- 172.21.15.173:0/3186161443 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f54f803c3c0 con 0x7f54fc100e40 2024-06-06T03:11:31.597 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:31.597 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":1,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:09:11.898452Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-06T03:11:31.597 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 1 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 msgr2=0x7f54e8036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 0x7f54e8036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f54ec006fd0 tx=0x7f54ec006db0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 msgr2=0x7f54fc10b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f54f8004820 tx=0x7f54f8009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 shutdown_connections 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f54e80349f0 0x7f54e8036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.599 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 --2- 172.21.15.173:0/3186161443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f54fc100e40 0x7f54fc10b480 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:31.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 >> 172.21.15.173:0/3186161443 conn(0x7f54fc070e10 msgr2=0x7f54fc0f9a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:31.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 shutdown_connections 2024-06-06T03:11:31.600 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:31.599+0000 7f5503472640 1 -- 172.21.15.173:0/3186161443 wait complete. 2024-06-06T03:11:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3186161443' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: Standby manager daemon smithi173.zwtggy started 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='mgr.? 172.21.15.173:0/753585993' entity='mgr.smithi173.zwtggy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.zwtggy/crt"}]: dispatch 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='mgr.? 172.21.15.173:0/753585993' entity='mgr.smithi173.zwtggy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='mgr.? 172.21.15.173:0/753585993' entity='mgr.smithi173.zwtggy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi173.zwtggy/key"}]: dispatch 2024-06-06T03:11:32.003 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:31 smithi153 ceph-mon[29300]: from='mgr.? 172.21.15.173:0/753585993' entity='mgr.smithi173.zwtggy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-06T03:11:32.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:31 smithi173 systemd[1]: Started Ceph mon.smithi173 for 0b80117e-23b2-11ef-bc9b-c7b262605968. 2024-06-06T03:11:33.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-06T03:11:33.537 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-06-06T03:11:33.797 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: Failed to load ceph-mgr modules: k8sevents 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mon.smithi153 calling monitor election 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mon.smithi173 calling monitor election 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mon.smithi153 is new leader, mons smithi153,smithi173 in quorum (ranks 0,1) 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi173=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: fsmap 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mgrmap e18: smithi153.gidtue(active, since 26s), standbys: smithi173.zwtggy 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-06T03:11:37.254 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:09:22.053083Z 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:09:31.545299Z 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:10:06.596864Z 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.255 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: Failed to load ceph-mgr modules: k8sevents 2024-06-06T03:11:37.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:11:37.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mon.smithi153 calling monitor election 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mon.smithi173 calling monitor election 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mon.smithi153 is new leader, mons smithi153,smithi173 in quorum (ranks 0,1) 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: monmap e2: 2 mons at {smithi153=[v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0],smithi173=[v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: fsmap 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: osdmap e5: 0 total, 0 up, 0 in 2024-06-06T03:11:37.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mgrmap e18: smithi153.gidtue(active, since 26s), standbys: smithi173.zwtggy 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:09:22.053083Z 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:09:31.545299Z 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: mgr module k8sevents crashed in daemon mgr.smithi153.gidtue on host smithi153 at 2024-06-06T03:10:06.596864Z 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:37.340 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 -- 172.21.15.173:0/4145751379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0d40034f0 msgr2=0x7fd0d4005970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 --2- 172.21.15.173:0/4145751379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0d40034f0 0x7fd0d4005970 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80faed0 tx=0x7fd0d800b040 comp rx=0 tx=0).stop 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 -- 172.21.15.173:0/4145751379 shutdown_connections 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 --2- 172.21.15.173:0/4145751379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0d40034f0 0x7fd0d4005970 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 -- 172.21.15.173:0/4145751379 >> 172.21.15.173:0/4145751379 conn(0x7fd0e80f6b40 msgr2=0x7fd0e80f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:38.183 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.182+0000 7fd0ef293640 1 -- 172.21.15.173:0/4145751379 shutdown_connections 2024-06-06T03:11:38.183 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:37 smithi173 ceph-mon[35650]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:38.183 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:37 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:11:38.184 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:37 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 -- 172.21.15.173:0/4145751379 wait complete. 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 Processor -- start 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 -- start start 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 0x7fd0e810e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e8116880 con 0x7fd0e810dcc0 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ef293640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0e81169c0 con 0x7fd0d40034f0 2024-06-06T03:11:38.186 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ed02f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.183+0000 7fd0ed02f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:54232/0 (socket says 172.21.15.173:54232) 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ed02f640 1 -- 172.21.15.173:0/2963770444 learned_addr learned my addr 172.21.15.173:0/2963770444 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ed02f640 1 -- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 msgr2=0x7fd0e810d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ed02f640 1 -- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 msgr2=0x7fd0e810d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ed02f640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ed02f640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ec82e640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 0x7fd0e810e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ec82e640 1 -- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 msgr2=0x7fd0e810d780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ec82e640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ec82e640 1 -- 172.21.15.173:0/2963770444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d8005040 con 0x7fd0e810dcc0 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ec82e640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 0x7fd0e810e0f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b580 tx=0x7fd0e40090f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e4011690 con 0x7fd0e810dcc0 2024-06-06T03:11:38.187 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e8116c20 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd0e4011cc0 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.184+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd0e400f340 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.185+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e8117110 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.186+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e80fc6e0 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.186+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fd0e40117f0 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.186+0000 7fd0ddffb640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 0x7fd0cc067960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.186+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd0e4081a50 con 0x7fd0e810dcc0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.187+0000 7fd0ed02f640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 0x7fd0cc067960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:38.188 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.187+0000 7fd0ed02f640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 0x7fd0cc067960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8007b40 tx=0x7fd0d800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:38.190 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.191+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd0e4050c40 con 0x7fd0e810dcc0 2024-06-06T03:11:38.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:37 smithi153 ceph-mon[29300]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:38.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:37 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:11:38.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:37 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:11:38.424 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.424+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0e804b480 con 0x7fd0e810dcc0 2024-06-06T03:11:38.425 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.425+0000 7fd0ddffb640 1 -- 172.21.15.173:0/2963770444 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7fd0e4015090 con 0x7fd0e810dcc0 2024-06-06T03:11:38.425 INFO:teuthology.orchestra.run.smithi173.stdout: 2024-06-06T03:11:38.425 INFO:teuthology.orchestra.run.smithi173.stdout:{"epoch":2,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","modified":"2024-06-06T03:11:31.789344Z","created":"2024-06-06T03:09:11.898452Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi153","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi173","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:3300","nonce":0},{"type":"v1","addr":"172.21.15.173:6789","nonce":0}]},"addr":"172.21.15.173:6789/0","public_addr":"172.21.15.173:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-06T03:11:38.425 INFO:teuthology.orchestra.run.smithi173.stderr:dumped monmap epoch 2 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 msgr2=0x7fd0cc067960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 0x7fd0cc067960 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8007b40 tx=0x7fd0d800b040 comp rx=0 tx=0).stop 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 msgr2=0x7fd0e810e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 0x7fd0e810e0f0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b580 tx=0x7fd0e40090f0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 shutdown_connections 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0e810dcc0 0x7fd0e810e0f0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd0cc0654b0 0x7fd0cc067960 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.427 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 --2- 172.21.15.173:0/2963770444 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd0d40034f0 0x7fd0e810d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:38.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 >> 172.21.15.173:0/2963770444 conn(0x7fd0e80f6b40 msgr2=0x7fd0e80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:38.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 shutdown_connections 2024-06-06T03:11:38.428 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:11:38.427+0000 7fd0ef293640 1 -- 172.21.15.173:0/2963770444 wait complete. 2024-06-06T03:11:38.974 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:38 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/2963770444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:38.998 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-06T03:11:38.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-06-06T03:11:39.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:38 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/2963770444' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-06T03:11:39.265 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-06T03:11:40.438 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T03:11:40.439 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: Updating smithi153:/etc/ceph/ceph.conf 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: Updating smithi173:/etc/ceph/ceph.conf 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T03:11:40.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: Updating smithi173:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: Updating smithi153:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: Reconfiguring daemon mon.smithi153 on smithi153 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:41.425 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: Updating smithi173:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: Updating smithi153:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/config/ceph.conf 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: Reconfiguring mon.smithi153 (unknown last config time)... 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: Reconfiguring daemon mon.smithi153 on smithi153 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 -- 172.21.15.153:0/687788805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 msgr2=0x7f40941015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 --2- 172.21.15.153:0/687788805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941015a0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f407c0048d0 tx=0x7f407c009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 -- 172.21.15.153:0/687788805 shutdown_connections 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 --2- 172.21.15.153:0/687788805 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 0x7f40940fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 --2- 172.21.15.153:0/687788805 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941015a0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:41.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 -- 172.21.15.153:0/687788805 >> 172.21.15.153:0/687788805 conn(0x7f40940f6dc0 msgr2=0x7f40940f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 -- 172.21.15.153:0/687788805 shutdown_connections 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.954+0000 7f4099a41640 1 -- 172.21.15.153:0/687788805 wait complete. 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 Processor -- start 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 -- start start 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 0x7f4094113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4094113ca0 con 0x7f40941011d0 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4099a41640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f409410d050 con 0x7f4094101b70 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38814/0 (socket says 172.21.15.153:38814) 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 -- 172.21.15.153:0/55163220 learned_addr learned my addr 172.21.15.153:0/55163220 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:11:41.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f40927fc640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 0x7f4094113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 -- 172.21.15.153:0/55163220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 msgr2=0x7f4094113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 0x7f4094113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.955+0000 7f4092ffd640 1 -- 172.21.15.153:0/55163220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c005040 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f4092ffd640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f407c003080 tx=0x7f407c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407c038070 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f407c008ee0 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409410d2d0 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f407c03d870 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.956+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f409410d790 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.957+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40940fc940 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.957+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f407c030020 con 0x7f40941011d0 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.957+0000 7f408b7fe640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 0x7f4080067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:41.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.958+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f407c0a1300 con 0x7f40941011d0 2024-06-06T03:11:41.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.958+0000 7f40927fc640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 0x7f4080067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:41.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.958+0000 7f40927fc640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 0x7f4080067850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f409410e990 tx=0x7f4084008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:41.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:41.962+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f407c070670 con 0x7f40941011d0 2024-06-06T03:11:42.133 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.133+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f409404b4b0 con 0x7f40941011d0 2024-06-06T03:11:42.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.133+0000 7f408b7fe640 1 -- 172.21.15.153:0/55163220 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f407c042100 con 0x7f40941011d0 2024-06-06T03:11:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:# minimal ceph.conf for 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:11:42.134 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-06-06T03:11:42.134 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 0b80117e-23b2-11ef-bc9b-c7b262605968 2024-06-06T03:11:42.134 INFO:teuthology.orchestra.run.smithi153.stdout: mon_host = [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] 2024-06-06T03:11:42.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 msgr2=0x7f4080067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:42.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 0x7f4080067850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f409410e990 tx=0x7f4084008040 comp rx=0 tx=0).stop 2024-06-06T03:11:42.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 msgr2=0x7f40941130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:42.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f407c003080 tx=0x7f407c009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:42.136 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 shutdown_connections 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f40800653a0 0x7f4080067850 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4094101b70 0x7f4094113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 --2- 172.21.15.153:0/55163220 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40941011d0 0x7f40941130d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.135+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 >> 172.21.15.153:0/55163220 conn(0x7f40940f6dc0 msgr2=0x7f4094065800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.136+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 shutdown_connections 2024-06-06T03:11:42.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:42.136+0000 7f4099a41640 1 -- 172.21.15.153:0/55163220 wait complete. 2024-06-06T03:11:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.gidtue", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T03:11:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T03:11:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi153.gidtue", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T03:11:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:42.818 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-06T03:11:42.818 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:11:42.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T03:11:42.847 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:11:42.847 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T03:11:42.914 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:11:42.914 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.conf 2024-06-06T03:11:42.947 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:11:42.947 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-06T03:11:43.015 INFO:tasks.cephadm:Deploying OSDs... 2024-06-06T03:11:43.016 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:11:43.016 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T03:11:43.032 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T03:11:43.032 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-06-06T03:11:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T03:11:43.087 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-06 03:11:04.664349581 +0000 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-06 03:05:31.564382629 +0000 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-06 03:05:31.564382629 +0000 2024-06-06T03:11:43.088 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-06 03:05:31.564382629 +0000 2024-06-06T03:11:43.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T03:11:43.151 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-06T03:11:43.151 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-06T03:11:43.151 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000189027 s, 2.7 MB/s 2024-06-06T03:11:43.152 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T03:11:43.208 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-06-06T03:11:43.263 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T03:11:43.263 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.263 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-06 03:11:04.665349564 +0000 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-06 03:05:31.919376711 +0000 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-06 03:05:31.919376711 +0000 2024-06-06T03:11:43.264 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-06 03:05:31.919376711 +0000 2024-06-06T03:11:43.264 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T03:11:43.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:43 smithi153 ceph-mon[29300]: Reconfiguring mgr.smithi153.gidtue (unknown last config time)... 2024-06-06T03:11:43.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:43 smithi153 ceph-mon[29300]: Reconfiguring daemon mgr.smithi153.gidtue on smithi153 2024-06-06T03:11:43.325 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:43 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/55163220' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:43.327 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-06T03:11:43.327 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-06T03:11:43.327 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000167076 s, 3.1 MB/s 2024-06-06T03:11:43.328 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T03:11:43.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:43 smithi173 ceph-mon[35650]: Reconfiguring mgr.smithi153.gidtue (unknown last config time)... 2024-06-06T03:11:43.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:43 smithi173 ceph-mon[35650]: Reconfiguring daemon mgr.smithi153.gidtue on smithi153 2024-06-06T03:11:43.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:43 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/55163220' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:43.384 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-06 03:11:04.665349564 +0000 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-06 03:05:32.249371211 +0000 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-06 03:05:32.249371211 +0000 2024-06-06T03:11:43.440 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-06 03:05:32.249371211 +0000 2024-06-06T03:11:43.441 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T03:11:43.503 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-06T03:11:43.503 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-06T03:11:43.503 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000204469 s, 2.5 MB/s 2024-06-06T03:11:43.505 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T03:11:43.561 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-06-06 03:11:04.666349548 +0000 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-06-06 03:05:32.596365428 +0000 2024-06-06T03:11:43.616 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-06-06 03:05:32.596365428 +0000 2024-06-06T03:11:43.617 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: 2024-06-06 03:05:32.596365428 +0000 2024-06-06T03:11:43.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T03:11:43.678 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-06-06T03:11:43.679 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-06-06T03:11:43.679 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000165166 s, 3.1 MB/s 2024-06-06T03:11:43.681 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T03:11:43.736 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:11:43.736 DEBUG:teuthology.orchestra.run.smithi173:> dd if=/scratch_devs of=/dev/stdout 2024-06-06T03:11:43.755 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-06T03:11:43.756 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_1 2024-06-06T03:11:43.814 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-06T03:11:43.814 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.814 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-06T03:11:43.814 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.815 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.815 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-06 03:11:30.094244896 +0000 2024-06-06T03:11:43.815 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-06 03:05:31.551478672 +0000 2024-06-06T03:11:43.815 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-06 03:05:31.551478672 +0000 2024-06-06T03:11:43.815 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-06 03:05:31.551478672 +0000 2024-06-06T03:11:43.815 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-06T03:11:43.881 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-06T03:11:43.881 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-06T03:11:43.881 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000234841 s, 2.2 MB/s 2024-06-06T03:11:43.883 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-06T03:11:43.940 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_2 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-06 03:11:30.095244875 +0000 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-06 03:05:31.886471250 +0000 2024-06-06T03:11:43.997 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-06 03:05:31.886471250 +0000 2024-06-06T03:11:43.998 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-06 03:05:31.886471250 +0000 2024-06-06T03:11:43.998 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-06T03:11:44.064 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-06T03:11:44.064 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-06T03:11:44.064 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000202097 s, 2.5 MB/s 2024-06-06T03:11:44.065 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-06T03:11:44.124 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_3 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-06 03:11:30.095244875 +0000 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-06 03:05:32.209464095 +0000 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-06 03:05:32.209464095 +0000 2024-06-06T03:11:44.182 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-06 03:05:32.209464095 +0000 2024-06-06T03:11:44.182 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-06T03:11:44.248 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-06T03:11:44.248 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-06T03:11:44.248 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000235337 s, 2.2 MB/s 2024-06-06T03:11:44.249 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-06T03:11:44.307 DEBUG:teuthology.orchestra.run.smithi173:> stat /dev/vg_nvme/lv_4 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Context: system_u:object_r:device_t:s0 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Access: 2024-06-06 03:11:30.095244875 +0000 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Modify: 2024-06-06 03:05:32.542456718 +0000 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout:Change: 2024-06-06 03:05:32.542456718 +0000 2024-06-06T03:11:44.364 INFO:teuthology.orchestra.run.smithi173.stdout: Birth: 2024-06-06 03:05:32.542456718 +0000 2024-06-06T03:11:44.365 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-06T03:11:44.427 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:44 smithi173 ceph-mon[35650]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:44.427 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:44 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:44.427 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:44 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:44.427 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:44 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:11:44.427 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:44 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:44.429 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records in 2024-06-06T03:11:44.429 INFO:teuthology.orchestra.run.smithi173.stderr:1+0 records out 2024-06-06T03:11:44.429 INFO:teuthology.orchestra.run.smithi173.stderr:512 bytes copied, 0.000184256 s, 2.8 MB/s 2024-06-06T03:11:44.431 DEBUG:teuthology.orchestra.run.smithi173:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-06T03:11:44.459 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:44 smithi153 ceph-mon[29300]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:44.460 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:44 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:44.460 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:44 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:44.460 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:44 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi153", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:11:44.460 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:44 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:44.488 INFO:tasks.cephadm:Deploying osd.0 on smithi153 with /dev/vg_nvme/lv_4... 2024-06-06T03:11:44.488 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T03:11:44.813 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:11:45.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:45 smithi153 ceph-mon[29300]: Reconfiguring crash.smithi153 (monmap changed)... 2024-06-06T03:11:45.163 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:45 smithi153 ceph-mon[29300]: Reconfiguring daemon crash.smithi153 on smithi153 2024-06-06T03:11:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:45 smithi173 ceph-mon[35650]: Reconfiguring crash.smithi153 (monmap changed)... 2024-06-06T03:11:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:45 smithi173 ceph-mon[35650]: Reconfiguring daemon crash.smithi153 on smithi153 2024-06-06T03:11:46.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:46 smithi153 ceph-mon[29300]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:46.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:46.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:46.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:46 smithi153 ceph-mon[29300]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-06-06T03:11:46.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:46 smithi153 ceph-mon[29300]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-06-06T03:11:46.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:46 smithi173 ceph-mon[35650]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:46.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:46.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:46.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:46 smithi173 ceph-mon[35650]: Reconfiguring alertmanager.smithi153 (dependencies changed)... 2024-06-06T03:11:46.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:46 smithi173 ceph-mon[35650]: Reconfiguring daemon alertmanager.smithi153 on smithi153 2024-06-06T03:11:48.062 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T03:11:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T03:11:48.174 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-06T03:11:48.174 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-06T03:11:48.174 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138012 s, 760 MB/s 2024-06-06T03:11:48.174 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-06T03:11:48.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:48 smithi153 ceph-mon[29300]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:48 smithi173 ceph-mon[35650]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:49.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_4 2024-06-06T03:11:49.992 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:11:50.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:50 smithi153 ceph-mon[29300]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:50.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:50 smithi173 ceph-mon[35650]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.628+0000 7f0288e0a640 1 -- 172.21.15.153:0/3941534638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840664f0 msgr2=0x7f02840feec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.628+0000 7f0288e0a640 1 --2- 172.21.15.153:0/3941534638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840664f0 0x7f02840feec0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f02740048d0 tx=0x7f0274009da0 comp rx=0 tx=0).stop 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- 172.21.15.153:0/3941534638 shutdown_connections 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 --2- 172.21.15.153:0/3941534638 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840ff400 0x7f02841017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 --2- 172.21.15.153:0/3941534638 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840664f0 0x7f02840feec0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- 172.21.15.153:0/3941534638 >> 172.21.15.153:0/3941534638 conn(0x7f02840f6970 msgr2=0x7f02840f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:11:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- 172.21.15.153:0/3941534638 shutdown_connections 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- 172.21.15.153:0/3941534638 wait complete. 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 Processor -- start 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- start start 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840664f0 0x7f028410cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028410dab0 con 0x7f02840ff400 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.629+0000 7f0288e0a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f028410e2d0 con 0x7f02840664f0 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47390/0 (socket says 172.21.15.153:47390) 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 -- 172.21.15.153:0/2706099784 learned_addr learned my addr 172.21.15.153:0/2706099784 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f028259c640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840664f0 0x7f028410cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 -- 172.21.15.153:0/2706099784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840664f0 msgr2=0x7f028410cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840664f0 0x7f028410cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 -- 172.21.15.153:0/2706099784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0274005040 con 0x7f02840ff400 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0281d9b640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0278002a50 tx=0x7f0278004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0278007c30 con 0x7f02840ff400 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f028410e560 con 0x7f02840ff400 2024-06-06T03:11:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.630+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f027800f040 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.631+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f027800dad0 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.631+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f028410eb00 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.632+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f027800dd40 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.632+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02840fa610 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.632+0000 7f0272ffd640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 0x7f026c067640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.632+0000 7f028259c640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 0x7f026c067640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.632+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0278081040 con 0x7f02840ff400 2024-06-06T03:11:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.633+0000 7f028259c640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 0x7f026c067640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0274000c00 tx=0x7f0274035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:11:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.636+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0278050350 con 0x7f02840ff400 2024-06-06T03:11:51.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:11:51.810+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f028410ed40 con 0x7f026c065190 2024-06-06T03:11:52.316 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:52 smithi153 ceph-mon[29300]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:52.316 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:52 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:11:52.316 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:52 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:11:52.316 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:52 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:52.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:52 smithi173 ceph-mon[35650]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:52.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:52 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:11:52.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:52 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:11:52.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:52 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:11:53.456 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:53 smithi153 ceph-mon[29300]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:11:53.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:53 smithi173 ceph-mon[35650]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:11:54.435 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:54 smithi153 ceph-mon[29300]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:54.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:54 smithi173 ceph-mon[35650]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:55.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:55 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:55.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:55 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:55.536 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:55 smithi153 ceph-mon[29300]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-06-06T03:11:55.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:55 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:55.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:55 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:11:55.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:55 smithi173 ceph-mon[35650]: Reconfiguring grafana.smithi153 (dependencies changed)... 2024-06-06T03:11:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-06-06T03:11:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3184422457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faf72683-4c0d-4d8b-9f4a-51826f225303"}]: dispatch 2024-06-06T03:11:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3184422457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faf72683-4c0d-4d8b-9f4a-51826f225303"}]': finished 2024-06-06T03:11:56.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T03:11:56.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:11:56.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:56.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:56 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2170688480' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: Reconfiguring daemon grafana.smithi153 on smithi153 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3184422457' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "faf72683-4c0d-4d8b-9f4a-51826f225303"}]: dispatch 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3184422457' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "faf72683-4c0d-4d8b-9f4a-51826f225303"}]': finished 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: osdmap e6: 1 total, 0 up, 1 in 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:11:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:56 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2170688480' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:11:58.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:11:58 smithi173 ceph-mon[35650]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:11:58.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:11:58 smithi153 ceph-mon[29300]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:00.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:00 smithi153 ceph-mon[29300]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:00.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:00 smithi173 ceph-mon[35650]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:02.535 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:02 smithi153 ceph-mon[29300]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:02.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:02 smithi173 ceph-mon[35650]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:04.498 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:04 smithi153 ceph-mon[29300]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:04.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:04 smithi173 ceph-mon[35650]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:05 smithi153 ceph-mon[29300]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:06.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:05 smithi153 ceph-mon[29300]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-06-06T03:12:06.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:05 smithi153 ceph-mon[29300]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-06-06T03:12:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:05 smithi173 ceph-mon[35650]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:05 smithi173 ceph-mon[35650]: Reconfiguring prometheus.smithi153 (dependencies changed)... 2024-06-06T03:12:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:05 smithi173 ceph-mon[35650]: Reconfiguring daemon prometheus.smithi153 on smithi153 2024-06-06T03:12:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:06 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T03:12:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:06 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:06 smithi153 ceph-mon[29300]: Deploying daemon osd.0 on smithi153 2024-06-06T03:12:07.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:06 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-06T03:12:07.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:06 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:07.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:06 smithi173 ceph-mon[35650]: Deploying daemon osd.0 on smithi153 2024-06-06T03:12:08.118 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:07 smithi153 ceph-mon[29300]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:08.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:07 smithi173 ceph-mon[35650]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:10.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:09 smithi153 ceph-mon[29300]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:10.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:09 smithi173 ceph-mon[35650]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:10.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:10.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:10.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:10.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:12:10.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:11.133 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:11.133 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:11.133 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:11.133 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi173", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-06T03:12:11.133 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:11.901 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:11 smithi153 ceph-mon[29300]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:11.901 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:11 smithi153 ceph-mon[29300]: Reconfiguring crash.smithi173 (monmap changed)... 2024-06-06T03:12:11.901 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:11 smithi153 ceph-mon[29300]: Reconfiguring daemon crash.smithi173 on smithi173 2024-06-06T03:12:12.135 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:11 smithi173 ceph-mon[35650]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:12.136 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:11 smithi173 ceph-mon[35650]: Reconfiguring crash.smithi173 (monmap changed)... 2024-06-06T03:12:12.136 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:11 smithi173 ceph-mon[35650]: Reconfiguring daemon crash.smithi173 on smithi173 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: Reconfiguring mgr.smithi173.zwtggy (monmap changed)... 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.zwtggy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:13.174 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:13 smithi153 ceph-mon[29300]: Reconfiguring daemon mgr.smithi173.zwtggy on smithi173 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: Reconfiguring mgr.smithi173.zwtggy (monmap changed)... 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi173.zwtggy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:13.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:13 smithi173 ceph-mon[35650]: Reconfiguring daemon mgr.smithi173.zwtggy on smithi173 2024-06-06T03:12:14.395 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: Reconfiguring mon.smithi173 (monmap changed)... 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:14.396 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:14 smithi153 ceph-mon[29300]: Reconfiguring daemon mon.smithi173 on smithi173 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: Reconfiguring mon.smithi173 (monmap changed)... 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:14 smithi173 ceph-mon[35650]: Reconfiguring daemon mon.smithi173 on smithi173 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.568 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:15.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:15 smithi153 ceph-mon[29300]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi153.front.sepia.ceph.com:9093"}]: dispatch 2024-06-06T03:12:15.749 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi153.front.sepia.ceph.com:3000"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi153.front.sepia.ceph.com:9095"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:15.750 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:15 smithi173 ceph-mon[35650]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:17.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:16 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:16 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:16 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:16 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:16 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:17.841 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:17 smithi153 ceph-mon[29300]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:18.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:17 smithi173 ceph-mon[35650]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:19.496 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 0 on host 'smithi153' 2024-06-06T03:12:19.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.496+0000 7f0272ffd640 1 -- 172.21.15.153:0/2706099784 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f028410ed40 con 0x7f026c065190 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 msgr2=0x7f026c067640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 0x7f026c067640 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0274000c00 tx=0x7f0274035040 comp rx=0 tx=0).stop 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 msgr2=0x7f028410d460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0278002a50 tx=0x7f0278004600 comp rx=0 tx=0).stop 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 shutdown_connections 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02840ff400 0x7f028410d460 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f026c065190 0x7f026c067640 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 --2- 172.21.15.153:0/2706099784 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f02840664f0 0x7f028410cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 >> 172.21.15.153:0/2706099784 conn(0x7f02840f6970 msgr2=0x7f02840f6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:12:19.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 shutdown_connections 2024-06-06T03:12:19.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:19.498+0000 7f0288e0a640 1 -- 172.21.15.153:0/2706099784 wait complete. 2024-06-06T03:12:20.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:19 smithi153 ceph-mon[29300]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:20.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:20.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:20.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:19 smithi153 ceph-mon[29300]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T03:12:20.042 DEBUG:teuthology.orchestra.run.smithi153:osd.0> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.0.service 2024-06-06T03:12:20.045 INFO:tasks.cephadm:Deploying osd.1 on smithi153 with /dev/vg_nvme/lv_3... 2024-06-06T03:12:20.045 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T03:12:20.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:19 smithi173 ceph-mon[35650]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:20.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:20.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:20.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:19 smithi173 ceph-mon[35650]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-06T03:12:20.473 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:12:21.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-mon[29300]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T03:12:21.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-mon[29300]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T03:12:21.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:21.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-mon[29300]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:12:21.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-mon[29300]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:21.753 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0[47221]: 2024-06-06T03:12:21.502+0000 7f66cc2a8640 -1 osd.0 0 waiting for initial osdmap 2024-06-06T03:12:21.753 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:12:21 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0[47221]: 2024-06-06T03:12:21.505+0000 7f66c68bc640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:12:21.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:21 smithi173 ceph-mon[35650]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-06T03:12:21.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:21 smithi173 ceph-mon[35650]: osdmap e7: 1 total, 0 up, 1 in 2024-06-06T03:12:21.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:21 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:21.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:21 smithi173 ceph-mon[35650]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:12:21.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:21 smithi173 ceph-mon[35650]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:22 smithi153 ceph-mon[29300]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:12:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:22 smithi153 ceph-mon[29300]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T03:12:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:22.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:22 smithi173 ceph-mon[35650]: from='osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:12:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:22 smithi173 ceph-mon[35650]: osdmap e8: 1 total, 0 up, 1 in 2024-06-06T03:12:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:23.510 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T03:12:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T03:12:23.625 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-06T03:12:23.626 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-06T03:12:23.626 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012919 s, 812 MB/s 2024-06-06T03:12:23.626 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] boot 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:23.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:23.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: osd.0 [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] boot 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: osdmap e9: 1 total, 1 up, 1 in 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:23.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:24 smithi153 ceph-mon[29300]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T03:12:24.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:24 smithi173 ceph-mon[35650]: osdmap e10: 1 total, 1 up, 1 in 2024-06-06T03:12:25.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_3 2024-06-06T03:12:25.650 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:12:25.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:25 smithi153 ceph-mon[29300]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:25.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:25 smithi173 ceph-mon[35650]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:26.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:26.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 -- 172.21.15.153:0/1674197770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 msgr2=0x7ff8700fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 --2- 172.21.15.153:0/1674197770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff8700fef40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff8640048d0 tx=0x7ff864009da0 comp rx=0 tx=0).stop 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 -- 172.21.15.153:0/1674197770 shutdown_connections 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 --2- 172.21.15.153:0/1674197770 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff870101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 --2- 172.21.15.153:0/1674197770 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff8700fef40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 -- 172.21.15.153:0/1674197770 >> 172.21.15.153:0/1674197770 conn(0x7ff8700f6970 msgr2=0x7ff8700f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 -- 172.21.15.153:0/1674197770 shutdown_connections 2024-06-06T03:12:27.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.858+0000 7ff8771ed640 1 -- 172.21.15.153:0/1674197770 wait complete. 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 Processor -- start 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 -- start start 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff87010d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87010dae0 con 0x7ff870066410 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff8771ed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87010e300 con 0x7ff8700ff480 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff874f89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff874f89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56014/0 (socket says 172.21.15.153:56014) 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff874f89640 1 -- 172.21.15.153:0/3524084080 learned_addr learned my addr 172.21.15.153:0/3524084080 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff86ffff640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff87010d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.859+0000 7ff874f89640 1 -- 172.21.15.153:0/3524084080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 msgr2=0x7ff87010d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:27.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff874f89640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff87010d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff874f89640 1 -- 172.21.15.153:0/3524084080 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff864005040 con 0x7ff870066410 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff86ffff640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff87010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff874f89640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff864000c00 tx=0x7ff864009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff864038070 con 0x7ff870066410 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87010e530 con 0x7ff870066410 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff864032970 con 0x7ff870066410 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff864033950 con 0x7ff870066410 2024-06-06T03:12:27.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.860+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87010ea70 con 0x7ff870066410 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.861+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8700fa610 con 0x7ff870066410 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.862+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7ff864030020 con 0x7ff870066410 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.863+0000 7ff86d7fa640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 0x7ff860067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.863+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7ff8640a1dd0 con 0x7ff870066410 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.863+0000 7ff86ffff640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 0x7ff860067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:27.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.863+0000 7ff86ffff640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 0x7ff860067850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff87010cd00 tx=0x7ff85c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:12:27.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:27.866+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff864070e40 con 0x7ff870066410 2024-06-06T03:12:28.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:28.051+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7ff87010ee60 con 0x7ff8600653a0 2024-06-06T03:12:28.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:28.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi153 2024-06-06T03:12:28.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:28.283 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi153 to 12864M 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi153 2024-06-06T03:12:28.284 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi153 to 12864M 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:28.285 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:29.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:12:29.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:12:29.142 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:12:29.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:29.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:29.143 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:29.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:30.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:29 smithi153 ceph-mon[29300]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:30.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:29 smithi173 ceph-mon[35650]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:32.061 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:31 smithi153 ceph-mon[29300]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:32.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:31 smithi173 ceph-mon[35650]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:33.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:32 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2161546673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad70997e-451a-4dfc-8b5a-2ce4a3851cb1"}]: dispatch 2024-06-06T03:12:33.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:32 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2161546673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad70997e-451a-4dfc-8b5a-2ce4a3851cb1"}]': finished 2024-06-06T03:12:33.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:32 smithi153 ceph-mon[29300]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T03:12:33.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:32 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:33.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:32 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2161546673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad70997e-451a-4dfc-8b5a-2ce4a3851cb1"}]: dispatch 2024-06-06T03:12:33.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:32 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2161546673' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad70997e-451a-4dfc-8b5a-2ce4a3851cb1"}]': finished 2024-06-06T03:12:33.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:32 smithi173 ceph-mon[35650]: osdmap e11: 2 total, 1 up, 2 in 2024-06-06T03:12:33.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:32 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3665186849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3665186849' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:36.264 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:36 smithi153 ceph-mon[29300]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:36.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:36 smithi173 ceph-mon[35650]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:38.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:38 smithi153 ceph-mon[29300]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:38.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:38 smithi173 ceph-mon[35650]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:40.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:40 smithi173 ceph-mon[35650]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:40.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T03:12:40.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:40.356 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:40 smithi153 ceph-mon[29300]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:40.357 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-06T03:12:40.357 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:41.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:41 smithi153 ceph-mon[29300]: Deploying daemon osd.1 on smithi153 2024-06-06T03:12:41.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:41.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:41 smithi173 ceph-mon[35650]: Deploying daemon osd.1 on smithi153 2024-06-06T03:12:41.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:42.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:42 smithi173 ceph-mon[35650]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:42.392 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:42 smithi153 ceph-mon[29300]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:44.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:44 smithi153 ceph-mon[29300]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:44.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:44 smithi173 ceph-mon[35650]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:45.331 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:45.331 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:45.331 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:45.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:45.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:45.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:46.286 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:46 smithi173 ceph-mon[35650]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:46.301 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:46 smithi153 ceph-mon[29300]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:47.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:47 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:47.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:47 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:47.718 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:47 smithi153 ceph-mon[29300]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:48.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:47 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:48.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:47 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:48.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:47 smithi173 ceph-mon[35650]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:48.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.563+0000 7ff86d7fa640 1 -- 172.21.15.153:0/3524084080 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff87010ee60 con 0x7ff8600653a0 2024-06-06T03:12:48.563 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 1 on host 'smithi153' 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 msgr2=0x7ff860067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 0x7ff860067850 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff87010cd00 tx=0x7ff85c009040 comp rx=0 tx=0).stop 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 msgr2=0x7ff87010cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff864000c00 tx=0x7ff864009da0 comp rx=0 tx=0).stop 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 shutdown_connections 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff8600653a0 0x7ff860067850 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff8700ff480 0x7ff87010d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 --2- 172.21.15.153:0/3524084080 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff870066410 0x7ff87010cf50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 >> 172.21.15.153:0/3524084080 conn(0x7ff8700f6970 msgr2=0x7ff8700f6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 shutdown_connections 2024-06-06T03:12:48.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:48.565+0000 7ff8771ed640 1 -- 172.21.15.153:0/3524084080 wait complete. 2024-06-06T03:12:49.185 DEBUG:teuthology.orchestra.run.smithi153:osd.1> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.1.service 2024-06-06T03:12:49.187 INFO:tasks.cephadm:Deploying osd.2 on smithi153 with /dev/vg_nvme/lv_2... 2024-06-06T03:12:49.187 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T03:12:49.591 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:12:49.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:49 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:49.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:49 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:49.814 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:49 smithi153 ceph-mon[29300]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:49.815 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:49 smithi153 ceph-mon[29300]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T03:12:49.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:49 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:49.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:49 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:49.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:49 smithi173 ceph-mon[35650]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:49.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:49 smithi173 ceph-mon[35650]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-06T03:12:50.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:50 smithi173 ceph-mon[35650]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T03:12:50.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:50 smithi173 ceph-mon[35650]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T03:12:50.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:50 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:50.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:50 smithi173 ceph-mon[35650]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:12:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-mon[29300]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-06T03:12:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-mon[29300]: osdmap e12: 2 total, 1 up, 2 in 2024-06-06T03:12:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:51.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-mon[29300]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:12:51.003 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1[54547]: 2024-06-06T03:12:50.572+0000 7fbf6770a640 -1 osd.1 0 waiting for initial osdmap 2024-06-06T03:12:51.003 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:12:50 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1[54547]: 2024-06-06T03:12:50.578+0000 7fbf60d3e640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:12:51.739 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:51 smithi173 ceph-mon[35650]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:12:51.739 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:51 smithi173 ceph-mon[35650]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T03:12:51.739 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:51 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:51.739 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:51 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:51.739 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:51 smithi173 ceph-mon[35650]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:51.741 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:51 smithi153 ceph-mon[29300]: from='osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:12:51.741 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:51 smithi153 ceph-mon[29300]: osdmap e13: 2 total, 1 up, 2 in 2024-06-06T03:12:51.741 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:51 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:51.741 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:51 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:51.742 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:51 smithi153 ceph-mon[29300]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-06T03:12:52.620 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T03:12:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T03:12:52.718 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10+0 records in 2024-06-06T03:12:52.718 INFO:teuthology.orchestra.run.smithi153.stderr:10+0 records out 2024-06-06T03:12:52.719 INFO:teuthology.orchestra.run.smithi153.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140024 s, 749 MB/s 2024-06-06T03:12:52.719 INFO:teuthology.orchestra.run.smithi153.stderr:--> Zapping successful for: 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] boot 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T03:12:52.830 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:52 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:52.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:12:52.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:12:52.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:52.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: osd.1 [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] boot 2024-06-06T03:12:52.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: osdmap e14: 2 total, 2 up, 2 in 2024-06-06T03:12:52.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:52 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-06T03:12:53.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:53 smithi153 ceph-mon[29300]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T03:12:53.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:53 smithi153 ceph-mon[29300]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:53.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:53 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:53.700 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:53 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:53.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi153:vg_nvme/lv_2 2024-06-06T03:12:53.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:53 smithi173 ceph-mon[35650]: osdmap e15: 2 total, 2 up, 2 in 2024-06-06T03:12:53.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:53 smithi173 ceph-mon[35650]: pgmap v54: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:53.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:53 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:53.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:53 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:54.472 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:12:56.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:55 smithi153 ceph-mon[29300]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:56.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:55 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:56.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:55 smithi173 ceph-mon[35650]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:56.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:55 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:12:56.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 -- 172.21.15.153:0/3899733741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40ff400 msgr2=0x7fb9f41017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 --2- 172.21.15.153:0/3899733741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40ff400 0x7fb9f41017e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc0048d0 tx=0x7fb9dc009da0 comp rx=0 tx=0).stop 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 -- 172.21.15.153:0/3899733741 shutdown_connections 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 --2- 172.21.15.153:0/3899733741 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f40ff400 0x7fb9f41017e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 --2- 172.21.15.153:0/3899733741 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f4066470 0x7fb9f40feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 -- 172.21.15.153:0/3899733741 >> 172.21.15.153:0/3899733741 conn(0x7fb9f40f6970 msgr2=0x7fb9f40f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 -- 172.21.15.153:0/3899733741 shutdown_connections 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.942+0000 7fb9f9334640 1 -- 172.21.15.153:0/3899733741 wait complete. 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 Processor -- start 2024-06-06T03:12:56.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 -- start start 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 0x7fb9f40fadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40fb3e0 con 0x7fb9f4066470 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.943+0000 7fb9f9334640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f40fb520 con 0x7fb9f40ff400 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59146/0 (socket says 172.21.15.153:59146) 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 -- 172.21.15.153:0/1618047549 learned_addr learned my addr 172.21.15.153:0/1618047549 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9ebfff640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 0x7fb9f40fadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 -- 172.21.15.153:0/1618047549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 msgr2=0x7fb9f40fadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 0x7fb9f40fadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 -- 172.21.15.153:0/1618047549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9dc005040 con 0x7fb9f4066470 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9ebfff640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 0x7fb9f40fadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:12:56.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f2ffd640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400b580 tx=0x7fb9e40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e400f570 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb9e400fba0 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f40fb800 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb9e4010dd0 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.944+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f4114f80 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.945+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f40fa610 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.946+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fb9e400f6d0 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.946+0000 7fb9eb7fe640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 0x7fb9e0067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.946+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fb9e4081a40 con 0x7fb9f4066470 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.946+0000 7fb9ebfff640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 0x7fb9e0067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.947+0000 7fb9ebfff640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 0x7fb9e0067850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009f80 tx=0x7fb9dc035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:12:56.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:56.951+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb9e4083e60 con 0x7fb9f4066470 2024-06-06T03:12:57.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:12:57.131+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9f40f84b0 con 0x7fb9e00653a0 2024-06-06T03:12:58.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:57 smithi153 ceph-mon[29300]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:58.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:57 smithi153 ceph-mon[29300]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:12:58.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:57 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:12:58.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:57 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:12:58.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:57 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:58.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:57 smithi173 ceph-mon[35650]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:12:58.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:57 smithi173 ceph-mon[35650]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi153:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:12:58.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:57 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:12:58.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:57 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:12:58.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:57 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi153 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi153 to 6432M 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.138 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:12:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi153 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi153 to 6432M 2024-06-06T03:12:59.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:12:59.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:12:59.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:12:59.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:12:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:00.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:00 smithi153 ceph-mon[29300]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:00.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:00 smithi173 ceph-mon[35650]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:01.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3466651834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d203663-86cb-4662-a7dd-2353c0a50101"}]: dispatch 2024-06-06T03:13:01.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3466651834' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d203663-86cb-4662-a7dd-2353c0a50101"}]': finished 2024-06-06T03:13:01.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:01 smithi153 ceph-mon[29300]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T03:13:01.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:01 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:01.223 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1693552840' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:01.389 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:01 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3466651834' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d203663-86cb-4662-a7dd-2353c0a50101"}]: dispatch 2024-06-06T03:13:01.390 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:01 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3466651834' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d203663-86cb-4662-a7dd-2353c0a50101"}]': finished 2024-06-06T03:13:01.390 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:01 smithi173 ceph-mon[35650]: osdmap e16: 3 total, 2 up, 3 in 2024-06-06T03:13:01.390 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:01 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:01.390 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:01 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1693552840' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:02.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:02 smithi153 ceph-mon[29300]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:02.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:02 smithi173 ceph-mon[35650]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:04.487 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:04.487 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:04.487 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:04.487 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:04.487 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:04.488 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:04.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:04.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:04.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:04.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:04.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:04.589 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:06.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:06 smithi153 ceph-mon[29300]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:06.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:06 smithi173 ceph-mon[35650]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:08.344 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:08 smithi153 ceph-mon[29300]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:08.344 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:08 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T03:13:08.344 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:08 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:08.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:08 smithi173 ceph-mon[35650]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:08.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:08 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-06T03:13:08.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:08 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:09.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:09 smithi153 ceph-mon[29300]: Deploying daemon osd.2 on smithi153 2024-06-06T03:13:09.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:09 smithi173 ceph-mon[35650]: Deploying daemon osd.2 on smithi153 2024-06-06T03:13:10.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:10 smithi153 ceph-mon[29300]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:10.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:10 smithi173 ceph-mon[35650]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:11.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:11.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:12.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:12 smithi153 ceph-mon[29300]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:12.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:12 smithi173 ceph-mon[35650]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:13.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:13.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:13.310 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:13 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:13.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:13.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:13.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:13 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:14.446 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:14 smithi153 ceph-mon[29300]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:14.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:14 smithi173 ceph-mon[35650]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:16.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:16 smithi153 ceph-mon[29300]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:16.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.563+0000 7fb9eb7fe640 1 -- 172.21.15.153:0/1618047549 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb9f40f84b0 con 0x7fb9e00653a0 2024-06-06T03:13:16.563 INFO:teuthology.orchestra.run.smithi153.stdout:Created osd(s) 2 on host 'smithi153' 2024-06-06T03:13:16.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.565+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 msgr2=0x7fb9e0067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:16.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.565+0000 7fb9f9334640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 0x7fb9e0067850 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc009f80 tx=0x7fb9dc035040 comp rx=0 tx=0).stop 2024-06-06T03:13:16.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.565+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 msgr2=0x7fb9f40fc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.565+0000 7fb9f9334640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb9e400b580 tx=0x7fb9e40091a0 comp rx=0 tx=0).stop 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 shutdown_connections 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb9e00653a0 0x7fb9e0067850 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb9f40ff400 0x7fb9f40fadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 --2- 172.21.15.153:0/1618047549 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f4066470 0x7fb9f40fc8d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 >> 172.21.15.153:0/1618047549 conn(0x7fb9f40f6970 msgr2=0x7fb9f40f6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 shutdown_connections 2024-06-06T03:13:16.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:13:16.566+0000 7fb9f9334640 1 -- 172.21.15.153:0/1618047549 wait complete. 2024-06-06T03:13:16.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:16 smithi173 ceph-mon[35650]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:17.303 DEBUG:teuthology.orchestra.run.smithi153:osd.2> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.2.service 2024-06-06T03:13:17.306 INFO:tasks.cephadm:Deploying osd.3 on smithi173 with /dev/vg_nvme/lv_4... 2024-06-06T03:13:17.306 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-06T03:13:17.566 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:13:17.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:17 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:17.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:17 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:17.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:17 smithi173 ceph-mon[35650]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:17.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:17 smithi173 ceph-mon[35650]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T03:13:17.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:17 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:17.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:17 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:17.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:17 smithi153 ceph-mon[29300]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:17.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:17 smithi153 ceph-mon[29300]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-06T03:13:18.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:18 smithi173 ceph-mon[35650]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T03:13:18.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:18 smithi173 ceph-mon[35650]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T03:13:18.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:18 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:18.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:18 smithi173 ceph-mon[35650]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:13:18.849 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-mon[29300]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-06T03:13:18.850 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-mon[29300]: osdmap e17: 3 total, 2 up, 3 in 2024-06-06T03:13:18.850 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:18.851 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-mon[29300]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]: dispatch 2024-06-06T03:13:18.851 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2[62236]: 2024-06-06T03:13:18.579+0000 7f9ed81ce640 -1 osd.2 0 waiting for initial osdmap 2024-06-06T03:13:18.851 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:18 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2[62236]: 2024-06-06T03:13:18.589+0000 7f9ed0fdf640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:13:19.222 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-06T03:13:19.313 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-06T03:13:19.327 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-06T03:13:19.328 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-06T03:13:19.328 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012264 s, 855 MB/s 2024-06-06T03:13:19.328 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-06T03:13:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:19 smithi153 ceph-mon[29300]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:13:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:19 smithi153 ceph-mon[29300]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T03:13:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:19 smithi153 ceph-mon[29300]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:19.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:19 smithi173 ceph-mon[35650]: from='osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi153", "root=default"]}]': finished 2024-06-06T03:13:19.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:19 smithi173 ceph-mon[35650]: osdmap e18: 3 total, 2 up, 3 in 2024-06-06T03:13:19.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:19.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:19.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:19 smithi173 ceph-mon[35650]: pgmap v70: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-06T03:13:19.932 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_4 2024-06-06T03:13:20.190 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] boot 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T03:13:20.618 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:20.619 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:20.619 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:20 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: osd.2 [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] boot 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: osdmap e19: 3 total, 3 up, 3 in 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:20.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:20 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:21.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- 172.21.15.173:0/4085956622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 msgr2=0x7f1eb80fda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:21.627 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/4085956622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb80fda00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f1ea80048d0 tx=0x7f1ea8009da0 comp rx=0 tx=0).stop 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- 172.21.15.173:0/4085956622 shutdown_connections 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/4085956622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb80fda00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/4085956622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb80fc6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- 172.21.15.173:0/4085956622 >> 172.21.15.173:0/4085956622 conn(0x7f1eb80f7a80 msgr2=0x7f1eb80f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- 172.21.15.173:0/4085956622 shutdown_connections 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- 172.21.15.173:0/4085956622 wait complete. 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 Processor -- start 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 -- start start 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.625+0000 7f1ebf15f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebf15f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebf15f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb810e170 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.628 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebf15f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb810e2e0 con 0x7f1eb80fd590 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1eaffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1eaffff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:47926/0 (socket says 172.21.15.173:47926) 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.173:45788/0 (socket says 172.21.15.173:45788) 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1eaffff640 1 -- 172.21.15.173:0/3645118153 learned_addr learned my addr 172.21.15.173:0/3645118153 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 -- 172.21.15.173:0/3645118153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 msgr2=0x7f1eb810d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 -- 172.21.15.173:0/3645118153 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea8005040 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1eaffff640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebcefb640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40027d0 tx=0x7f1eb4009f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb4007a20 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.629 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb810e5c0 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1eb400f040 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1eb4009490 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.630 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.626+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb810eb60 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.628+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb8105640 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.628+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f1eb4009750 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.628+0000 7f1ead7fa640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 0x7f1ea0067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.628+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7f1eb40815a0 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.628+0000 7f1eaffff640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 0x7f1ea0067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:21.631 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.629+0000 7f1eaffff640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 0x7f1ea0067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8004540 tx=0x7f1ea8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:13:21.634 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.633+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1eb404cdf0 con 0x7f1eb80fc2c0 2024-06-06T03:13:21.646 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:21 smithi153 ceph-mon[29300]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T03:13:21.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:21 smithi153 ceph-mon[29300]: pgmap v73: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:21.647 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:21 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-06T03:13:21.806 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:21.805+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f1eb810ed80 con 0x7f1ea00653a0 2024-06-06T03:13:22.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:21 smithi173 ceph-mon[35650]: osdmap e20: 3 total, 3 up, 3 in 2024-06-06T03:13:22.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:21 smithi173 ceph-mon[35650]: pgmap v73: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:22.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:21 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-06T03:13:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-06T03:13:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T03:13:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T03:13:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:13:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:13:22.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:13:22.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:22 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: osdmap e21: 3 total, 3 up, 3 in 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:13:22.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:13:22.839 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:22 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:23.800 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T03:13:23.800 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi153 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi153 to 4288M 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:23.801 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.802 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:23 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:23.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66953]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T03:13:23.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66953]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:23.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66953]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:23.959 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66953]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:23.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-06T03:13:23.959 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: osdmap e22: 3 total, 3 up, 3 in 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: pgmap v76: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi153 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi153 to 4288M 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:23.960 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:23 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:24.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66972]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T03:13:24.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66972]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:24.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66972]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:24.243 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:13:23 smithi153 sudo[66972]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:24.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[66977]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T03:13:24.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[66977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:24.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[66977]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:24.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[66977]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:24.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67064]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T03:13:24.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67064]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:24.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67064]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:24.762 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67064]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:25.016 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67087]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T03:13:25.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67087]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:25.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67087]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:25.017 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:24 smithi153 sudo[67087]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:25.270 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67090]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T03:13:25.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67090]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:25.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67090]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:25.271 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67090]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/3414805676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]: dispatch 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]: dispatch 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]': finished 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:25.271 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1092478750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:25.272 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67138]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/3414805676' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]: dispatch 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]: dispatch 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1410b23-185c-457c-a009-e90924e8dbaf"}]': finished 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: osdmap e23: 4 total, 3 up, 4 in 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:25.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/1092478750' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:25.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67138]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:25.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67138]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:25.530 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67138]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:25.543 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:25.544+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1eb4050530 con 0x7f1eb80fc2c0 2024-06-06T03:13:25.794 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 sudo[39509]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-06T03:13:25.794 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 sudo[39509]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67168]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T03:13:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67168]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67168]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:26.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:25 smithi153 sudo[67168]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:26.051 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 sudo[39509]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:26.051 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:25 smithi173 sudo[39509]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:26.326 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:26.326 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T03:13:26.326 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T03:13:26.326 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:13:26.327 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:13:26.327 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: mgrmap e19: smithi153.gidtue(active, since 2m), standbys: smithi173.zwtggy 2024-06-06T03:13:26.327 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:13:26.327 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:13:26.327 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 sudo[39513]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 sudo[39513]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 sudo[39513]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 sudo[39513]: pam_unix(sudo:session): session closed for user root 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: pgmap v78: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: mgrmap e19: smithi153.gidtue(active, since 2m), standbys: smithi173.zwtggy 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi153"}]: dispatch 2024-06-06T03:13:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "mon metadata", "id": "smithi173"}]: dispatch 2024-06-06T03:13:26.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:27.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:27 smithi153 ceph-mon[29300]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T03:13:27.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:27 smithi153 ceph-mon[29300]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T03:13:27.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:27 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:27.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:27 smithi173 ceph-mon[35650]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-06T03:13:27.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:27 smithi173 ceph-mon[35650]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-06T03:13:27.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:27 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: pgmap v79: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:30.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:30 smithi173 ceph-mon[35650]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:30.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:30 smithi153 ceph-mon[29300]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:31.389 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T03:13:31.389 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:31.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-06T03:13:31.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:32.358 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:32 smithi173 ceph-mon[35650]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:32.358 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:32 smithi173 ceph-mon[35650]: Deploying daemon osd.3 on smithi173 2024-06-06T03:13:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:32 smithi153 ceph-mon[29300]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:32.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:32 smithi153 ceph-mon[29300]: Deploying daemon osd.3 on smithi173 2024-06-06T03:13:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:34 smithi173 ceph-mon[35650]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:34 smithi153 ceph-mon[29300]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:36.188 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:36 smithi173 ceph-mon[35650]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:36.189 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:36.189 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:36.189 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:36 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:36 smithi153 ceph-mon[29300]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:36 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:38.127 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.127+0000 7f1ead7fa640 1 -- 172.21.15.173:0/3645118153 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1eb810ed80 con 0x7f1ea00653a0 2024-06-06T03:13:38.127 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 3 on host 'smithi173' 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 msgr2=0x7f1ea0067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 0x7f1ea0067850 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8004540 tx=0x7f1ea8035040 comp rx=0 tx=0).stop 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 msgr2=0x7f1eb810cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1eb40027d0 tx=0x7f1eb4009f70 comp rx=0 tx=0).stop 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 shutdown_connections 2024-06-06T03:13:38.129 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1ea00653a0 0x7f1ea0067850 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1eb80fd590 0x7f1eb810d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 --2- 172.21.15.173:0/3645118153 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1eb80fc2c0 0x7f1eb810cf00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.129+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 >> 172.21.15.173:0/3645118153 conn(0x7f1eb80f7a80 msgr2=0x7f1eb80f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:13:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.130+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 shutdown_connections 2024-06-06T03:13:38.130 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:38.130+0000 7f1ebf15f640 1 -- 172.21.15.173:0/3645118153 wait complete. 2024-06-06T03:13:38.277 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:38 smithi153 ceph-mon[29300]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:38.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:38 smithi173 ceph-mon[35650]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:39.042 DEBUG:teuthology.orchestra.run.smithi173:osd.3> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.3.service 2024-06-06T03:13:39.045 INFO:tasks.cephadm:Deploying osd.4 on smithi173 with /dev/vg_nvme/lv_3... 2024-06-06T03:13:39.045 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-06T03:13:39.266 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:39 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:39.266 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:39 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:39.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:39 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:39.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:39 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:39.588 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:13:39 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:13:39.335+0000 7f09426872c0 -1 osd.3 0 log_to_monitors true 2024-06-06T03:13:39.795 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:13:40.224 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:40 smithi173 ceph-mon[35650]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:40.224 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:40 smithi173 ceph-mon[35650]: from='osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T03:13:40.225 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:40 smithi173 ceph-mon[35650]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T03:13:40.225 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:40.225 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:40 smithi153 ceph-mon[29300]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:40 smithi153 ceph-mon[29300]: from='osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T03:13:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:40 smithi153 ceph-mon[29300]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-06T03:13:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:41.482 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:13:41.146+0000 7f093ddf9640 -1 osd.3 0 waiting for initial osdmap 2024-06-06T03:13:41.482 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:13:41.151+0000 7f0938c30640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:13:41.482 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T03:13:41.483 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T03:13:41.483 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:41.483 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: from='osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:13:41.483 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:13:41.483 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: osdmap e24: 4 total, 3 up, 4 in 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: from='osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:13:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:41.577 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-06T03:13:41.665 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-06T03:13:41.680 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-06T03:13:41.680 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-06T03:13:41.680 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124013 s, 846 MB/s 2024-06-06T03:13:41.680 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-06T03:13:42.253 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_3 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] boot 2024-06-06T03:13:42.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T03:13:42.503 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:42 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: osdmap e25: 4 total, 3 up, 4 in 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: osd.3 [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] boot 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: osdmap e26: 4 total, 4 up, 4 in 2024-06-06T03:13:42.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:42 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-06T03:13:42.915 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:13:43.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:43 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:13:43.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:43 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:13:43.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:43 smithi173 ceph-mon[35650]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T03:13:43.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:43 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:13:43.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:43 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:13:43.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:43 smithi153 ceph-mon[29300]: osdmap e27: 4 total, 4 up, 4 in 2024-06-06T03:13:44.434 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:44 smithi173 ceph-mon[35650]: pgmap v90: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:44.434 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:44 smithi173 ceph-mon[35650]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T03:13:44.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:44 smithi153 ceph-mon[29300]: pgmap v90: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:44.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:44 smithi153 ceph-mon[29300]: osdmap e28: 4 total, 4 up, 4 in 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 -- 172.21.15.173:0/2015732337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fb180 msgr2=0x7f93a40fb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 --2- 172.21.15.173:0/2015732337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fb180 0x7f93a40fb590 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93940048d0 tx=0x7f9394009da0 comp rx=0 tx=0).stop 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 -- 172.21.15.173:0/2015732337 shutdown_connections 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 --2- 172.21.15.173:0/2015732337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fc450 0x7f93a40fc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 --2- 172.21.15.173:0/2015732337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fb180 0x7f93a40fb590 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 -- 172.21.15.173:0/2015732337 >> 172.21.15.173:0/2015732337 conn(0x7f93a40f6960 msgr2=0x7f93a40f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 -- 172.21.15.173:0/2015732337 shutdown_connections 2024-06-06T03:13:44.821 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 -- 172.21.15.173:0/2015732337 wait complete. 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.810+0000 7f93a9b57640 1 Processor -- start 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a9b57640 1 -- start start 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a9b57640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 0x7f93a410fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a9b57640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a9b57640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4066250 con 0x7f93a40fb180 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a9b57640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a40663c0 con 0x7f93a40fc450 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:59612/0 (socket says 172.21.15.173:59612) 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 -- 172.21.15.173:0/3486981870 learned_addr learned my addr 172.21.15.173:0/3486981870 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a37fe640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 0x7f93a410fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 -- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 msgr2=0x7f93a410fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:13:44.822 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 0x7f93a410fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a2ffd640 1 -- 172.21.15.173:0/3486981870 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394005040 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.811+0000 7f93a37fe640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 0x7f93a410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f93a2ffd640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9398002a50 tx=0x7f9398004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9398007c30 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f939800f040 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f939800db50 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a41126d0 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.812+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a4112bf0 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.813+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a40fa5e0 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.813+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9398009070 con 0x7f93a40fc450 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.814+0000 7f938bfff640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 0x7f938c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:13:44.823 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.814+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f9398081a70 con 0x7f93a40fc450 2024-06-06T03:13:44.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.814+0000 7f93a37fe640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 0x7f938c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:13:44.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.815+0000 7f93a37fe640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 0x7f938c0678e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9394000c00 tx=0x7f9394035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:13:44.824 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:44.818+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9398050280 con 0x7f93a40fc450 2024-06-06T03:13:45.000 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:13:45.001+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f93a4100770 con 0x7f938c065430 2024-06-06T03:13:45.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T03:13:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:13:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:13:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:13:45.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:45 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: osdmap e29: 4 total, 4 up, 4 in 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: pgmap v94: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:13:45.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:45 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:46.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T03:13:46.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi173 2024-06-06T03:13:46.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.639 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi173 to 15936M 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:46.640 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:46 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: osdmap e30: 4 total, 4 up, 4 in 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi173 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:13:46.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi173 to 15936M 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:13:46.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:46 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:47.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:47 smithi173 ceph-mon[35650]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:47.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:47 smithi153 ceph-mon[29300]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/45978277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]: dispatch 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]: dispatch 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]': finished 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:13:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:48 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/1244095744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:48.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/45978277' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]: dispatch 2024-06-06T03:13:48.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]: dispatch 2024-06-06T03:13:48.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6d58503b-bf9e-48b6-a984-cf8476ce3f6d"}]': finished 2024-06-06T03:13:48.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: osdmap e31: 5 total, 4 up, 5 in 2024-06-06T03:13:48.703 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:13:48.704 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:48 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1244095744' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:13:49.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:49 smithi153 ceph-mon[29300]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-06-06T03:13:49.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:49 smithi173 ceph-mon[35650]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-06-06T03:13:51.972 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:51 smithi173 ceph-mon[35650]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-06-06T03:13:52.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:51 smithi153 ceph-mon[29300]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-06-06T03:13:54.048 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:53 smithi173 ceph-mon[35650]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T03:13:54.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:53 smithi153 ceph-mon[29300]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-06T03:13:55.087 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:54 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T03:13:55.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:54 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:55.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:54 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-06T03:13:55.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:54 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:13:56.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:55 smithi173 ceph-mon[35650]: Deploying daemon osd.4 on smithi173 2024-06-06T03:13:56.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:55 smithi173 ceph-mon[35650]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-06-06T03:13:56.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:55 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:55 smithi153 ceph-mon[29300]: Deploying daemon osd.4 on smithi173 2024-06-06T03:13:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:55 smithi153 ceph-mon[29300]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-06-06T03:13:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:55 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:13:58.077 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:57 smithi173 ceph-mon[35650]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-06T03:13:58.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:57 smithi153 ceph-mon[29300]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-06T03:13:59.950 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:59 smithi173 ceph-mon[35650]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:13:59.950 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:13:59.951 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:13:59.951 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:13:59 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:00.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:59 smithi153 ceph-mon[29300]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:00.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:00.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:00.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:13:59 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:02.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:01 smithi173 ceph-mon[35650]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:02.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:01 smithi153 ceph-mon[29300]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:02.860 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.860+0000 7f938bfff640 1 -- 172.21.15.173:0/3486981870 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f93a4100770 con 0x7f938c065430 2024-06-06T03:14:02.860 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 4 on host 'smithi173' 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 msgr2=0x7f938c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 0x7f938c0678e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9394000c00 tx=0x7f9394035040 comp rx=0 tx=0).stop 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 msgr2=0x7f93a4112130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9398002a50 tx=0x7f9398004600 comp rx=0 tx=0).stop 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 shutdown_connections 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f938c065430 0x7f938c0678e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f93a40fc450 0x7f93a4112130 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:02.862 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 --2- 172.21.15.173:0/3486981870 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f93a40fb180 0x7f93a410fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:02.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 >> 172.21.15.173:0/3486981870 conn(0x7f93a40f6960 msgr2=0x7f93a40f8010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:02.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 shutdown_connections 2024-06-06T03:14:02.863 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:02.862+0000 7f93a9b57640 1 -- 172.21.15.173:0/3486981870 wait complete. 2024-06-06T03:14:03.396 DEBUG:teuthology.orchestra.run.smithi173:osd.4> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.4.service 2024-06-06T03:14:03.398 INFO:tasks.cephadm:Deploying osd.5 on smithi173 with /dev/vg_nvme/lv_2... 2024-06-06T03:14:03.398 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-06T03:14:03.774 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:14:04.106 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:03 smithi173 ceph-mon[35650]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:04.106 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:03 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:04.107 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:03 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:04.107 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:03 smithi173 ceph-mon[35650]: from='osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T03:14:04.107 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:03 smithi173 ceph-mon[35650]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T03:14:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:03 smithi153 ceph-mon[29300]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:03 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:03 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:03 smithi153 ceph-mon[29300]: from='osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T03:14:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:03 smithi153 ceph-mon[29300]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-06T03:14:05.072 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-mon[35650]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T03:14:05.073 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-mon[35650]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T03:14:05.073 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:05.073 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-mon[35650]: from='osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:05.073 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-mon[35650]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:05.073 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:14:04.874+0000 7f2bb5e17640 -1 osd.4 0 waiting for initial osdmap 2024-06-06T03:14:05.073 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:14:04 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:14:04.884+0000 7f2baec4a640 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:14:05.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:04 smithi153 ceph-mon[29300]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-06T03:14:05.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:04 smithi153 ceph-mon[29300]: osdmap e32: 5 total, 4 up, 5 in 2024-06-06T03:14:05.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:04 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:05.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:04 smithi153 ceph-mon[29300]: from='osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:05.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:04 smithi153 ceph-mon[29300]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:05.492 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-06T03:14:05.580 INFO:teuthology.orchestra.run.smithi173.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-06T03:14:05.594 INFO:teuthology.orchestra.run.smithi173.stderr: stderr: 10+0 records in 2024-06-06T03:14:05.594 INFO:teuthology.orchestra.run.smithi173.stderr:10+0 records out 2024-06-06T03:14:05.594 INFO:teuthology.orchestra.run.smithi173.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012565 s, 835 MB/s 2024-06-06T03:14:05.594 INFO:teuthology.orchestra.run.smithi173.stderr:--> Zapping successful for: 2024-06-06T03:14:06.232 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:06.232 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:14:06.233 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T03:14:06.233 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:06.233 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:06.233 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:06.233 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:05 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: osdmap e33: 5 total, 4 up, 5 in 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:05 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:07.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:06 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:14:07.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:06 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:14:07.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:06 smithi173 ceph-mon[35650]: osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] boot 2024-06-06T03:14:07.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:06 smithi173 ceph-mon[35650]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T03:14:07.088 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:06 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:06 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:14:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:06 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:14:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:06 smithi153 ceph-mon[29300]: osd.4 [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] boot 2024-06-06T03:14:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:06 smithi153 ceph-mon[29300]: osdmap e34: 5 total, 5 up, 5 in 2024-06-06T03:14:07.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:06 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-06T03:14:07.343 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi173:vg_nvme/lv_2 2024-06-06T03:14:07.847 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:14:08.227 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:07 smithi173 ceph-mon[35650]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:08.227 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:07 smithi173 ceph-mon[35650]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T03:14:08.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:07 smithi153 ceph-mon[29300]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-06T03:14:08.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:07 smithi153 ceph-mon[29300]: osdmap e35: 5 total, 5 up, 5 in 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi173 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi173 to 7968M 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:09.327 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:09.328 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:09.328 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:09 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi173 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi173 to 7968M 2024-06-06T03:14:09.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:09.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:09 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 -- 172.21.15.173:0/4167078212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fc280 msgr2=0x7f46600fc690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 --2- 172.21.15.173:0/4167078212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fc280 0x7f46600fc690 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f46500048d0 tx=0x7f4650009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 -- 172.21.15.173:0/4167078212 shutdown_connections 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 --2- 172.21.15.173:0/4167078212 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fd550 0x7f46600fd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 --2- 172.21.15.173:0/4167078212 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fc280 0x7f46600fc690 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 -- 172.21.15.173:0/4167078212 >> 172.21.15.173:0/4167078212 conn(0x7f46600f7a40 msgr2=0x7f46600f9e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 -- 172.21.15.173:0/4167078212 shutdown_connections 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.776+0000 7f4665c53640 1 -- 172.21.15.173:0/4167078212 wait complete. 2024-06-06T03:14:09.797 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 Processor -- start 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 -- start start 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fc280 0x7f466010cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466010e2b0 con 0x7f46600fc280 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4665c53640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466010e420 con 0x7f46600fd550 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:46160/0 (socket says 172.21.15.173:46160) 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 -- 172.21.15.173:0/2916950587 learned_addr learned my addr 172.21.15.173:0/2916950587 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f465f7fe640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fc280 0x7f466010cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 -- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fc280 msgr2=0x7f466010cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fc280 0x7f466010cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 -- 172.21.15.173:0/2916950587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650005040 con 0x7f46600fd550 2024-06-06T03:14:09.798 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.777+0000 7f4657fff640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f464800b790 tx=0x7f46480091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.778+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f464800c700 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.778+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f464800cd30 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.778+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4648013370 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.778+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466010e700 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.778+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466010ec20 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.779+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46600fb6e0 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.779+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4648012070 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.780+0000 7f465cff9640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 0x7f464c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.780+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f4648081cb0 con 0x7f46600fd550 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.780+0000 7f465f7fe640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 0x7f464c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.780+0000 7f465f7fe640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 0x7f464c0678e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4650007ac0 tx=0x7f4650035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:09.799 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.784+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4648050310 con 0x7f46600fd550 2024-06-06T03:14:09.954 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:09.954+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f466010eea0 con 0x7f464c065430 2024-06-06T03:14:10.583 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:10 smithi173 ceph-mon[35650]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:10.583 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:14:10.584 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:14:10.584 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:10.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:10 smithi153 ceph-mon[29300]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:10.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-06T03:14:10.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-06T03:14:10.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:11.463 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:11 smithi173 ceph-mon[35650]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:14:11.463 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:11.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:11 smithi153 ceph-mon[29300]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi173:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-06T03:14:11.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:12.533 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:12.533 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/238463427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]: dispatch 2024-06-06T03:14:12.533 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]: dispatch 2024-06-06T03:14:12.533 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]': finished 2024-06-06T03:14:12.533 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T03:14:12.534 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:12 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/238463427' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]: dispatch 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]: dispatch 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c32699eb-9044-467a-b18e-f627e5991fa8"}]': finished 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: osdmap e36: 6 total, 5 up, 6 in 2024-06-06T03:14:12.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:12 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:13.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:13 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/1317285597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:14:13.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:13 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/1317285597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-06T03:14:14.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:14 smithi173 ceph-mon[35650]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:14.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:14 smithi153 ceph-mon[29300]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:16.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:16 smithi173 ceph-mon[35650]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:16.611 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:16 smithi153 ceph-mon[29300]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:18.544 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:18 smithi153 ceph-mon[29300]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:18.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:18 smithi173 ceph-mon[35650]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:19.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T03:14:19.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:19 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-06T03:14:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:19 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:20.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:20 smithi173 ceph-mon[35650]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:20.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:20 smithi173 ceph-mon[35650]: Deploying daemon osd.5 on smithi173 2024-06-06T03:14:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:20 smithi153 ceph-mon[29300]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:20 smithi153 ceph-mon[29300]: Deploying daemon osd.5 on smithi173 2024-06-06T03:14:22.480 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:22 smithi173 ceph-mon[35650]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:22 smithi153 ceph-mon[29300]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:24 smithi173 ceph-mon[35650]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:24 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:24 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:24 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:24 smithi153 ceph-mon[29300]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:24.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:24 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:26.524 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:26 smithi173 ceph-mon[35650]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:26.525 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:26.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:26 smithi153 ceph-mon[29300]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:26.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:28.235 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.235+0000 7f465cff9640 1 -- 172.21.15.173:0/2916950587 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f466010eea0 con 0x7f464c065430 2024-06-06T03:14:28.235 INFO:teuthology.orchestra.run.smithi173.stdout:Created osd(s) 5 on host 'smithi173' 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 msgr2=0x7f464c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 0x7f464c0678e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4650007ac0 tx=0x7f4650035040 comp rx=0 tx=0).stop 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 msgr2=0x7f466010d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f464800b790 tx=0x7f46480091d0 comp rx=0 tx=0).stop 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 shutdown_connections 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f464c065430 0x7f464c0678e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f46600fd550 0x7f466010d440 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 --2- 172.21.15.173:0/2916950587 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46600fc280 0x7f466010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 >> 172.21.15.173:0/2916950587 conn(0x7f46600f7a40 msgr2=0x7f46600f9110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 shutdown_connections 2024-06-06T03:14:28.237 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:28.237+0000 7f4665c53640 1 -- 172.21.15.173:0/2916950587 wait complete. 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T03:14:28.537 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T03:14:28.538 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:28.538 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.538 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:28 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T03:14:28.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-06T03:14:28.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:14:28.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.753 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:28 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:28.826 DEBUG:teuthology.orchestra.run.smithi173:osd.5> sudo journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.5.service 2024-06-06T03:14:28.829 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-06T03:14:28.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T03:14:29.081 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: from='osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:30.233 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:29 smithi153 ceph-mon[29300]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: osdmap e37: 6 total, 5 up, 6 in 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: from='osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:30.268 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-mon[35650]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]: dispatch 2024-06-06T03:14:30.269 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:14:29.986+0000 7ff07ed4c640 -1 osd.5 0 waiting for initial osdmap 2024-06-06T03:14:30.269 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:14:29 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:14:29.995+0000 7ff07835e640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-06T03:14:30.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- 172.21.15.153:0/1416454421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 msgr2=0x7f33f40fb5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 --2- 172.21.15.153:0/1416454421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f40fb5a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f33e80048d0 tx=0x7f33e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- 172.21.15.153:0/1416454421 shutdown_connections 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 --2- 172.21.15.153:0/1416454421 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f40fc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 --2- 172.21.15.153:0/1416454421 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f40fb5a0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- 172.21.15.153:0/1416454421 >> 172.21.15.153:0/1416454421 conn(0x7f33f40f6970 msgr2=0x7f33f40f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- 172.21.15.153:0/1416454421 shutdown_connections 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- 172.21.15.153:0/1416454421 wait complete. 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 Processor -- start 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.808+0000 7f33fad11640 1 -- start start 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:30.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f410f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f410fc80 con 0x7f33f40fb190 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4110450 con 0x7f33f40fc460 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48990/0 (socket says 172.21.15.153:48990) 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 -- 172.21.15.153:0/4157494164 learned_addr learned my addr 172.21.15.153:0/4157494164 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f3fff640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f410f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 -- 172.21.15.153:0/4157494164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 msgr2=0x7f33f410f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f410f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 -- 172.21.15.153:0/4157494164 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e8005040 con 0x7f33f40fb190 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f3fff640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f410f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f8aad640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009f80 tx=0x7f33e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e8038070 con 0x7f33f40fb190 2024-06-06T03:14:30.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33e8032970 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4110680 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f33e8033860 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.809+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f4110bc0 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.810+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f40fa5f0 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.811+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f33e8030020 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.811+0000 7f33f17fa640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 0x7f33e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.811+0000 7f33f3fff640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 0x7f33e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.811+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4354+0+0 (secure 0 0 0) 0x7f33e80a22d0 con 0x7f33f40fb190 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.812+0000 7f33f3fff640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 0x7f33e40678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009fd0 tx=0x7f33e0009e50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:30.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.816+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f33e8070890 con 0x7f33f40fb190 2024-06-06T03:14:30.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.982+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f33f404b480 con 0x7f33f40fb190 2024-06-06T03:14:30.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.982+0000 7f33f17fa640 1 -- 172.21.15.153:0/4157494164 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v38) v1 ==== 74+0+130 (secure 0 0 0) 0x7f33e8073eb0 con 0x7f33f40fb190 2024-06-06T03:14:30.982 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 msgr2=0x7f33e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 0x7f33e40678e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f33e0009fd0 tx=0x7f33e0009e50 comp rx=0 tx=0).stop 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 msgr2=0x7f33f410f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f33e8009f80 tx=0x7f33e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 shutdown_connections 2024-06-06T03:14:30.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f33e4065430 0x7f33e40678e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f33f40fc460 0x7f33f410f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 --2- 172.21.15.153:0/4157494164 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33f40fb190 0x7f33f410f0f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:30.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.984+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 >> 172.21.15.153:0/4157494164 conn(0x7f33f40f6970 msgr2=0x7f33f4065250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:30.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.985+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 shutdown_connections 2024-06-06T03:14:30.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:30.985+0000 7f33fad11640 1 -- 172.21.15.153:0/4157494164 wait complete. 2024-06-06T03:14:31.014 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:30 smithi153 ceph-mon[29300]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:14:31.014 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:30 smithi153 ceph-mon[29300]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T03:14:31.015 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:30 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:31.015 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:30 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:31.015 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:30 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4157494164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T03:14:31.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:30 smithi173 ceph-mon[35650]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi173", "root=default"]}]': finished 2024-06-06T03:14:31.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:30 smithi173 ceph-mon[35650]: osdmap e38: 6 total, 5 up, 6 in 2024-06-06T03:14:31.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:30 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:31.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:30 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:31.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:30 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4157494164' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T03:14:31.761 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":38,"num_osds":6,"num_up_osds":5,"osd_up_since":1717643645,"num_in_osds":6,"osd_in_since":1717643652,"num_remapped_pgs":0} 2024-06-06T03:14:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: purged_snaps scrub starts 2024-06-06T03:14:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: purged_snaps scrub ok 2024-06-06T03:14:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] boot 2024-06-06T03:14:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:32.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:31 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: purged_snaps scrub starts 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: purged_snaps scrub ok 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: osd.5 [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] boot 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: osdmap e39: 6 total, 6 up, 6 in 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-06T03:14:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:14:32.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:31 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:14:32.762 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-06-06T03:14:33.025 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:33.148 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:32 smithi153 ceph-mon[29300]: Detected new or changed devices on smithi173 2024-06-06T03:14:33.148 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:32 smithi153 ceph-mon[29300]: Adjusting osd_memory_target on smithi173 to 5312M 2024-06-06T03:14:33.148 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:32 smithi153 ceph-mon[29300]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T03:14:33.342 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:32 smithi173 ceph-mon[35650]: Detected new or changed devices on smithi173 2024-06-06T03:14:33.343 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:32 smithi173 ceph-mon[35650]: Adjusting osd_memory_target on smithi173 to 5312M 2024-06-06T03:14:33.343 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:32 smithi173 ceph-mon[35650]: osdmap e40: 6 total, 6 up, 6 in 2024-06-06T03:14:34.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:33 smithi153 ceph-mon[29300]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:33 smithi173 ceph-mon[35650]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 -- 172.21.15.153:0/572756339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 msgr2=0x7f59200fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 --2- 172.21.15.153:0/572756339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f59200fc8f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f59100048d0 tx=0x7f5910009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 -- 172.21.15.153:0/572756339 shutdown_connections 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 --2- 172.21.15.153:0/572756339 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f59200fc8f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 --2- 172.21.15.153:0/572756339 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 0x7f59200fb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 -- 172.21.15.153:0/572756339 >> 172.21.15.153:0/572756339 conn(0x7f59200f6970 msgr2=0x7f59200f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 -- 172.21.15.153:0/572756339 shutdown_connections 2024-06-06T03:14:34.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.705+0000 7f5928377640 1 -- 172.21.15.153:0/572756339 wait complete. 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 Processor -- start 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 -- start start 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 0x7f592010ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592010da40 con 0x7f59200fc480 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5928377640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592010e260 con 0x7f59200fb1b0 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49002/0 (socket says 172.21.15.153:49002) 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 -- 172.21.15.153:0/577447135 learned_addr learned my addr 172.21.15.153:0/577447135 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5926113640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 0x7f592010ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:34.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 -- 172.21.15.153:0/577447135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 msgr2=0x7f592010ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 0x7f592010ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 -- 172.21.15.153:0/577447135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910005040 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.706+0000 7f5925912640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5910009f80 tx=0x7f5910009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.707+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5910038070 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.707+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5910032970 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.707+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592010e490 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.707+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5910033860 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.707+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592010e9d0 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.708+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59200fa610 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.708+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5910030020 con 0x7f59200fc480 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.708+0000 7f5916ffd640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 0x7f590c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:34.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.708+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f59100a2330 con 0x7f59200fc480 2024-06-06T03:14:34.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.708+0000 7f5926113640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 0x7f590c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:34.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.709+0000 7f5926113640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 0x7f590c0678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f591c004ed0 tx=0x7f591c00b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:34.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.713+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5910070890 con 0x7f59200fc480 2024-06-06T03:14:34.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.878+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f5920103df0 con 0x7f59200fc480 2024-06-06T03:14:34.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.878+0000 7f5916ffd640 1 -- 172.21.15.153:0/577447135 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7f5910073eb0 con 0x7f59200fc480 2024-06-06T03:14:34.879 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:34.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 msgr2=0x7f590c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:34.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 0x7f590c0678e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f591c004ed0 tx=0x7f591c00b040 comp rx=0 tx=0).stop 2024-06-06T03:14:34.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 msgr2=0x7f592010d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5910009f80 tx=0x7f5910009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 shutdown_connections 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59200fc480 0x7f592010d3f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f590c065430 0x7f590c0678e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 --2- 172.21.15.153:0/577447135 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59200fb1b0 0x7f592010ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 >> 172.21.15.153:0/577447135 conn(0x7f59200f6970 msgr2=0x7f59200f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 shutdown_connections 2024-06-06T03:14:34.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:34.881+0000 7f5928377640 1 -- 172.21.15.153:0/577447135 wait complete. 2024-06-06T03:14:35.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:35 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/577447135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T03:14:35.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:35 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/577447135' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-06T03:14:35.498 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1717643670,"num_in_osds":6,"osd_in_since":1717643652,"num_remapped_pgs":0} 2024-06-06T03:14:35.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T03:14:35.752 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:36.193 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:36 smithi153 ceph-mon[29300]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:36.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:36 smithi173 ceph-mon[35650]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- 172.21.15.153:0/2849544942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f81011c0 msgr2=0x7f32f8101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 --2- 172.21.15.153:0/2849544942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f81011c0 0x7f32f8101590 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f32ec0048d0 tx=0x7f32ec009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- 172.21.15.153:0/2849544942 shutdown_connections 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 --2- 172.21.15.153:0/2849544942 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f8101b60 0x7f32f80fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 --2- 172.21.15.153:0/2849544942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f81011c0 0x7f32f8101590 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- 172.21.15.153:0/2849544942 >> 172.21.15.153:0/2849544942 conn(0x7f32f80f6db0 msgr2=0x7f32f80f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:37.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- 172.21.15.153:0/2849544942 shutdown_connections 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- 172.21.15.153:0/2849544942 wait complete. 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 Processor -- start 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.435+0000 7f32feda5640 1 -- start start 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32feda5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f81011c0 0x7f32f810df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32feda5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32feda5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f810eb10 con 0x7f32f8101b60 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32feda5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f810ec80 con 0x7f32f81011c0 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49016/0 (socket says 172.21.15.153:49016) 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 -- 172.21.15.153:0/3168623593 learned_addr learned my addr 172.21.15.153:0/3168623593 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32fcb41640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f81011c0 0x7f32f810df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 -- 172.21.15.153:0/3168623593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f81011c0 msgr2=0x7f32f810df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:37.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f81011c0 0x7f32f810df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 -- 172.21.15.153:0/3168623593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec005040 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f7fff640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f32e4002a50 tx=0x7f32e4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e4007c30 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f32e400f040 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.436+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f8116ff0 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.437+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32e400dbe0 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.437+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f8117500 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.437+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f80fc930 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.438+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f32e400dd40 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.438+0000 7f32f57fa640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 0x7f32e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.439+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f32e40823d0 con 0x7f32f8101b60 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.439+0000 7f32fcb41640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 0x7f32e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:37.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.439+0000 7f32fcb41640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 0x7f32e80678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000c00 tx=0x7f32ec035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:37.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.443+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f32e4050880 con 0x7f32f8101b60 2024-06-06T03:14:37.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.608+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32f80fa9b0 con 0x7f32f8101b60 2024-06-06T03:14:37.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.609+0000 7f32f57fa640 1 -- 172.21.15.153:0/3168623593 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11131 (secure 0 0 0) 0x7f32e4053ea0 con 0x7f32f8101b60 2024-06-06T03:14:37.610 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:37.611 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","created":"2024-06-06T03:09:14.685479+0000","modified":"2024-06-06T03:14:31.988451+0000","last_up_change":"2024-06-06T03:14:30.983590+0000","last_in_change":"2024-06-06T03:14:12.198580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T03:13:21.002067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"faf72683-4c0d-4d8b-9f4a-51826f225303","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6803","nonce":2356787937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6805","nonce":2356787937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6809","nonce":2356787937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6807","nonce":2356787937}]},"public_addr":"172.21.15.153:6803/2356787937","cluster_addr":"172.21.15.153:6805/2356787937","heartbeat_back_addr":"172.21.15.153:6809/2356787937","heartbeat_front_addr":"172.21.15.153:6807/2356787937","state":["exists","up"]},{"osd":1,"uuid":"ad70997e-451a-4dfc-8b5a-2ce4a3851cb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6811","nonce":4085830578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6813","nonce":4085830578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6817","nonce":4085830578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6815","nonce":4085830578}]},"public_addr":"172.21.15.153:6811/4085830578","cluster_addr":"172.21.15.153:6813/4085830578","heartbeat_back_addr":"172.21.15.153:6817/4085830578","heartbeat_front_addr":"172.21.15.153:6815/4085830578","state":["exists","up"]},{"osd":2,"uuid":"4d203663-86cb-4662-a7dd-2353c0a50101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6819","nonce":858759998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6821","nonce":858759998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6825","nonce":858759998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6823","nonce":858759998}]},"public_addr":"172.21.15.153:6819/858759998","cluster_addr":"172.21.15.153:6821/858759998","heartbeat_back_addr":"172.21.15.153:6825/858759998","heartbeat_front_addr":"172.21.15.153:6823/858759998","state":["exists","up"]},{"osd":3,"uuid":"e1410b23-185c-457c-a009-e90924e8dbaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6801","nonce":1222772148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6803","nonce":1222772148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6807","nonce":1222772148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6805","nonce":1222772148}]},"public_addr":"172.21.15.173:6801/1222772148","cluster_addr":"172.21.15.173:6803/1222772148","heartbeat_back_addr":"172.21.15.173:6807/1222772148","heartbeat_front_addr":"172.21.15.173:6805/1222772148","state":["exists","up"]},{"osd":4,"uuid":"6d58503b-bf9e-48b6-a984-cf8476ce3f6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6809","nonce":2520823922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6811","nonce":2520823922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6815","nonce":2520823922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6813","nonce":2520823922}]},"public_addr":"172.21.15.173:6809/2520823922","cluster_addr":"172.21.15.173:6811/2520823922","heartbeat_back_addr":"172.21.15.173:6815/2520823922","heartbeat_front_addr":"172.21.15.173:6813/2520823922","state":["exists","up"]},{"osd":5,"uuid":"c32699eb-9044-467a-b18e-f627e5991fa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6817","nonce":2901238167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6819","nonce":2901238167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6823","nonce":2901238167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6821","nonce":2901238167}]},"public_addr":"172.21.15.173:6817/2901238167","cluster_addr":"172.21.15.173:6819/2901238167","heartbeat_back_addr":"172.21.15.173:6823/2901238167","heartbeat_front_addr":"172.21.15.173:6821/2901238167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:20.508700+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:49.716707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:17.772334+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:40.350532+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:04.456105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:29.082829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2262799474":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/372809069":"2024-06-07T03:11:10.698615+0000","172.21.15.153:6800/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/161244006":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6800/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6801/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/3807092861":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/295427621":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/1168792425":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6800/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/2789596196":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/3471374002":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/2811436119":"2024-06-07T03:11:10.698615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T03:14:37.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.611+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 msgr2=0x7f32e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.611+0000 7f32feda5640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 0x7f32e80678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f32ec000c00 tx=0x7f32ec035040 comp rx=0 tx=0).stop 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.611+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 msgr2=0x7f32f810e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.611+0000 7f32feda5640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f32e4002a50 tx=0x7f32e4004600 comp rx=0 tx=0).stop 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 shutdown_connections 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f32f8101b60 0x7f32f810e4c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f32e8065430 0x7f32e80678e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 --2- 172.21.15.153:0/3168623593 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f32f81011c0 0x7f32f810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:37.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 >> 172.21.15.153:0/3168623593 conn(0x7f32f80f6db0 msgr2=0x7f32f80f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:37.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 shutdown_connections 2024-06-06T03:14:37.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:37.612+0000 7f32feda5640 1 -- 172.21.15.153:0/3168623593 wait complete. 2024-06-06T03:14:38.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:38 smithi153 ceph-mon[29300]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:38.237 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:38 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3168623593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:38.263 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-06T03:13:21.002067+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-06T03:14:38.263 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-06T03:14:38.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:38 smithi173 ceph-mon[35650]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:38.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:38 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3168623593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:38.520 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:40.236 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:40 smithi153 ceph-mon[29300]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 -- 172.21.15.153:0/1173053839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 msgr2=0x7fd4481015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 --2- 172.21.15.153:0/1173053839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481015a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd43c0048d0 tx=0x7fd43c009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 -- 172.21.15.153:0/1173053839 shutdown_connections 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 --2- 172.21.15.153:0/1173053839 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 0x7fd4480fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 --2- 172.21.15.153:0/1173053839 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481015a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 -- 172.21.15.153:0/1173053839 >> 172.21.15.153:0/1173053839 conn(0x7fd4480f6dc0 msgr2=0x7fd4480f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:40.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 -- 172.21.15.153:0/1173053839 shutdown_connections 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.235+0000 7fd44f433640 1 -- 172.21.15.153:0/1173053839 wait complete. 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 Processor -- start 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 -- start start 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 0x7fd448113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd448113ca0 con 0x7fd4481011d0 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44f433640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44810d050 con 0x7fd448101b70 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44d1cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.236+0000 7fd44d1cf640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50958/0 (socket says 172.21.15.153:50958) 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44d1cf640 1 -- 172.21.15.153:0/3609980598 learned_addr learned my addr 172.21.15.153:0/3609980598 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44c9ce640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 0x7fd448113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44d1cf640 1 -- 172.21.15.153:0/3609980598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 msgr2=0x7fd448113610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44d1cf640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 0x7fd448113610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44d1cf640 1 -- 172.21.15.153:0/3609980598 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c005040 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44d1cf640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd43c003080 tx=0x7fd43c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd43c038070 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd43c008ee0 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44810d2d0 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd43c03d870 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.237+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44810d790 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.238+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4480fc940 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.239+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd43c030020 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.239+0000 7fd439ffb640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 0x7fd4340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.239+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd43c0a21d0 con 0x7fd4481011d0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.239+0000 7fd44c9ce640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 0x7fd4340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.240+0000 7fd44c9ce640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 0x7fd4340678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd44810e990 tx=0x7fd444008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:40.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.243+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd43c070700 con 0x7fd4481011d0 2024-06-06T03:14:40.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:40 smithi173 ceph-mon[35650]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:40.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.408+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd44804b4b0 con 0x7fd4481011d0 2024-06-06T03:14:40.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.408+0000 7fd439ffb640 1 -- 172.21.15.153:0/3609980598 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd43c073d20 con 0x7fd4481011d0 2024-06-06T03:14:40.409 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 msgr2=0x7fd4340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 0x7fd4340678e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd44810e990 tx=0x7fd444008040 comp rx=0 tx=0).stop 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 msgr2=0x7fd4481130d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd43c003080 tx=0x7fd43c009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 shutdown_connections 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd434065430 0x7fd4340678e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd448101b70 0x7fd448113610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 --2- 172.21.15.153:0/3609980598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4481011d0 0x7fd4481130d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 >> 172.21.15.153:0/3609980598 conn(0x7fd4480f6dc0 msgr2=0x7fd448065800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 shutdown_connections 2024-06-06T03:14:40.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:40.410+0000 7fd44f433640 1 -- 172.21.15.153:0/3609980598 wait complete. 2024-06-06T03:14:40.992 INFO:tasks.cephadm:Setting up client nodes... 2024-06-06T03:14:40.993 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T03:14:41.244 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:41.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:41 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3609980598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T03:14:41.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:41.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:41 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3609980598' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-06T03:14:41.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:42.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:42 smithi173 ceph-mon[35650]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:42.449 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:42 smithi153 ceph-mon[29300]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 -- 172.21.15.153:0/352624337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 msgr2=0x7fc6240fd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 --2- 172.21.15.153:0/352624337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc6240fd9d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc6180048d0 tx=0x7fc618009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 -- 172.21.15.153:0/352624337 shutdown_connections 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 --2- 172.21.15.153:0/352624337 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc6240fd9d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 --2- 172.21.15.153:0/352624337 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc6240fc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 -- 172.21.15.153:0/352624337 >> 172.21.15.153:0/352624337 conn(0x7fc6240f7a70 msgr2=0x7fc6240f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 -- 172.21.15.153:0/352624337 shutdown_connections 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.972+0000 7fc6297ec640 1 -- 172.21.15.153:0/352624337 wait complete. 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 Processor -- start 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 -- start start 2024-06-06T03:14:42.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc62410ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62410da40 con 0x7fc6240fd560 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6297ec640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc62410e260 con 0x7fc6240fc290 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6227fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6227fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50970/0 (socket says 172.21.15.153:50970) 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.973+0000 7fc6227fc640 1 -- 172.21.15.153:0/3137551848 learned_addr learned my addr 172.21.15.153:0/3137551848 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc622ffd640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc62410ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6227fc640 1 -- 172.21.15.153:0/3137551848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 msgr2=0x7fc62410ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6227fc640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc62410ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6227fc640 1 -- 172.21.15.153:0/3137551848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618005040 con 0x7fc6240fd560 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc622ffd640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc62410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6227fc640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc618009f80 tx=0x7fc618009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:42.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc618038070 con 0x7fc6240fd560 2024-06-06T03:14:42.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc618032970 con 0x7fc6240fd560 2024-06-06T03:14:42.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc62410e490 con 0x7fc6240fd560 2024-06-06T03:14:42.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc618033860 con 0x7fc6240fd560 2024-06-06T03:14:42.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.974+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc62410e9d0 con 0x7fc6240fd560 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.975+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6240fb6f0 con 0x7fc6240fd560 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.976+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc618030020 con 0x7fc6240fd560 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.976+0000 7fc60b7fe640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 0x7fc60c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.976+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc6180a23f0 con 0x7fc6240fd560 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.976+0000 7fc622ffd640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 0x7fc60c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:42.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.976+0000 7fc622ffd640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 0x7fc60c0678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc614004d80 tx=0x7fc61400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:42.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:42.980+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc6180708a0 con 0x7fc6240fd560 2024-06-06T03:14:43.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.206+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc62404b480 con 0x7fc6240fd560 2024-06-06T03:14:43.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.210+0000 7fc60b7fe640 1 -- 172.21.15.153:0/3137551848 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc6180410f0 con 0x7fc6240fd560 2024-06-06T03:14:43.210 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-06-06T03:14:43.210 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQCjKWFmb8duDBAA9AOl6PaJt7ofuJRcLEIKeQ== 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 msgr2=0x7fc60c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 0x7fc60c0678e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc614004d80 tx=0x7fc61400b040 comp rx=0 tx=0).stop 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 msgr2=0x7fc62410d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc618009f80 tx=0x7fc618009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 shutdown_connections 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc6240fd560 0x7fc62410d3f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc60c065430 0x7fc60c0678e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 --2- 172.21.15.153:0/3137551848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc6240fc290 0x7fc62410ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 >> 172.21.15.153:0/3137551848 conn(0x7fc6240f7a70 msgr2=0x7fc6240f9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 shutdown_connections 2024-06-06T03:14:43.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:43.212+0000 7fc6297ec640 1 -- 172.21.15.153:0/3137551848 wait complete. 2024-06-06T03:14:43.999 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-06-06T03:14:43.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-06T03:14:43.999 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-06T03:14:44.036 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-06T03:14:44.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:44 smithi153 ceph-mon[29300]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:44.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:44 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3137551848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:44.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:44 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3137551848' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T03:14:44.299 INFO:teuthology.orchestra.run.smithi173.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi173/config 2024-06-06T03:14:44.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:44 smithi173 ceph-mon[35650]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:44.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:44 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3137551848' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:44.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:44 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3137551848' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.706+0000 7fa99d2cb640 1 -- 172.21.15.173:0/1630642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fc2c0 msgr2=0x7fa9980fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.706+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/1630642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fc2c0 0x7fa9980fc6d0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa988009c50 tx=0x7fa988004d10 comp rx=0 tx=0).stop 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- 172.21.15.173:0/1630642405 shutdown_connections 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/1630642405 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fd590 0x7fa9980fda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/1630642405 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fc2c0 0x7fa9980fc6d0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- 172.21.15.173:0/1630642405 >> 172.21.15.173:0/1630642405 conn(0x7fa9980f7a60 msgr2=0x7fa9980f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- 172.21.15.173:0/1630642405 shutdown_connections 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- 172.21.15.173:0/1630642405 wait complete. 2024-06-06T03:14:45.708 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 Processor -- start 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- start start 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fc2c0 0x7fa99810cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99810da50 con 0x7fa9980fc2c0 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.707+0000 7fa99d2cb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99810e270 con 0x7fa9980fd590 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.173:3300/0 says I am v2:172.21.15.173:52656/0 (socket says 172.21.15.173:52656) 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 -- 172.21.15.173:0/2703371912 learned_addr learned my addr 172.21.15.173:0/2703371912 (peer_addr_for_me v2:172.21.15.173:0/0) 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa996ffd640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fc2c0 0x7fa99810cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 -- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fc2c0 msgr2=0x7fa99810cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fc2c0 0x7fa99810cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 -- 172.21.15.173:0/2703371912 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9880098b0 con 0x7fa9980fd590 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa9967fc640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa98000eb70 tx=0x7fa98000b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:45.709 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa980007c30 con 0x7fa9980fd590 2024-06-06T03:14:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99810e4a0 con 0x7fa9980fd590 2024-06-06T03:14:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa98000f040 con 0x7fa9980fd590 2024-06-06T03:14:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa9800095b0 con 0x7fa9980fd590 2024-06-06T03:14:45.710 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.708+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99810e9c0 con 0x7fa9980fd590 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.709+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9980fb720 con 0x7fa9980fd590 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.710+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa980009730 con 0x7fa9980fd590 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.710+0000 7fa98f7fe640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 0x7fa9840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.710+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa980082140 con 0x7fa9980fd590 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.710+0000 7fa996ffd640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 0x7fa9840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:45.711 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.711+0000 7fa996ffd640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 0x7fa9840678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa988000f80 tx=0x7fa988004d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:45.713 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.714+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa9800506a0 con 0x7fa9980fd590 2024-06-06T03:14:45.938 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.939+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fa99804b480 con 0x7fa9980fd590 2024-06-06T03:14:45.943 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.944+0000 7fa98f7fe640 1 -- 172.21.15.173:0/2703371912 <== mon.1 v2:172.21.15.173:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fa980053cc0 con 0x7fa9980fd590 2024-06-06T03:14:45.944 INFO:teuthology.orchestra.run.smithi173.stdout:[client.1] 2024-06-06T03:14:45.944 INFO:teuthology.orchestra.run.smithi173.stdout: key = AQClKWFmU7MYOBAAsw2hNdlUw3q/0zX6HD8E+A== 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 msgr2=0x7fa9840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 0x7fa9840678e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa988000f80 tx=0x7fa988004d10 comp rx=0 tx=0).stop 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 msgr2=0x7fa99810d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa98000eb70 tx=0x7fa98000b040 comp rx=0 tx=0).stop 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 shutdown_connections 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa984065430 0x7fa9840678e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9980fd590 0x7fa99810d400 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.946 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 --2- 172.21.15.173:0/2703371912 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9980fc2c0 0x7fa99810cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 >> 172.21.15.173:0/2703371912 conn(0x7fa9980f7a60 msgr2=0x7fa9980f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 shutdown_connections 2024-06-06T03:14:45.947 INFO:teuthology.orchestra.run.smithi173.stderr:2024-06-06T03:14:45.946+0000 7fa99d2cb640 1 -- 172.21.15.173:0/2703371912 wait complete. 2024-06-06T03:14:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:46 smithi173 ceph-mon[35650]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:46 smithi173 ceph-mon[35650]: from='client.? 172.21.15.173:0/2703371912' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:46 smithi173 ceph-mon[35650]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:46 smithi173 ceph-mon[35650]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T03:14:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:46 smithi153 ceph-mon[29300]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:46 smithi153 ceph-mon[29300]: from='client.? 172.21.15.173:0/2703371912' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:46 smithi153 ceph-mon[29300]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-06T03:14:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:46 smithi153 ceph-mon[29300]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-06T03:14:46.558 DEBUG:teuthology.orchestra.run.smithi173:> set -ex 2024-06-06T03:14:46.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-06T03:14:46.558 DEBUG:teuthology.orchestra.run.smithi173:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-06T03:14:46.595 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-06T03:14:46.595 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-06T03:14:46.595 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-06-06T03:14:46.858 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:48.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:48 smithi153 ceph-mon[29300]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:48.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:48 smithi173 ceph-mon[35650]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:48.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- 172.21.15.153:0/2032784703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fdea0 msgr2=0x7ff3c80fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:48.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2032784703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fdea0 0x7ff3c80fe310 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0048d0 tx=0x7ff3bc009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- 172.21.15.153:0/2032784703 shutdown_connections 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2032784703 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c80fdea0 0x7ff3c80fe310 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2032784703 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c8065030 0x7ff3c8065400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- 172.21.15.153:0/2032784703 >> 172.21.15.153:0/2032784703 conn(0x7ff3c8072110 msgr2=0x7ff3c8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- 172.21.15.153:0/2032784703 shutdown_connections 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- 172.21.15.153:0/2032784703 wait complete. 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 Processor -- start 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- start start 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 0x7ff3c81135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.613+0000 7ff3d0627640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c8113bb0 con 0x7ff3c8065030 2024-06-06T03:14:48.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3d0627640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c810d040 con 0x7ff3c80fdea0 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:50980/0 (socket says 172.21.15.153:50980) 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 -- 172.21.15.153:0/2473453167 learned_addr learned my addr 172.21.15.153:0/2473453167 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3cdbc2640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 0x7ff3c81135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 -- 172.21.15.153:0/2473453167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 msgr2=0x7ff3c81135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 0x7ff3c81135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 -- 172.21.15.153:0/2473453167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc005040 con 0x7ff3c8065030 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3cdbc2640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 0x7ff3c81135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3ce3c3640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800ece0 tx=0x7ff3b800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b800ee40 con 0x7ff3c8065030 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c810d320 con 0x7ff3c8065030 2024-06-06T03:14:48.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff3b800f040 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff3b8009750 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.614+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c810d840 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.615+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c80fda40 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.616+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff3b8007890 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.616+0000 7ff3c6ffd640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 0x7ff3b00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.616+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff3b8082d60 con 0x7ff3c8065030 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.616+0000 7ff3cdbc2640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 0x7ff3b00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.616+0000 7ff3cdbc2640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 0x7ff3b00678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff3c810e930 tx=0x7ff3bc035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:48.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.620+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff3b8051210 con 0x7ff3c8065030 2024-06-06T03:14:48.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.830+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff3c804b480 con 0x7ff3c8065030 2024-06-06T03:14:48.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.833+0000 7ff3c6ffd640 1 -- 172.21.15.153:0/2473453167 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161455 (secure 0 0 0) 0x7ff3b8058e60 con 0x7ff3c8065030 2024-06-06T03:14:48.835 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:48.836 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 msgr2=0x7ff3b00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 0x7ff3b00678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff3c810e930 tx=0x7ff3bc035040 comp rx=0 tx=0).stop 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 msgr2=0x7ff3c8113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800ece0 tx=0x7ff3b800b040 comp rx=0 tx=0).stop 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 shutdown_connections 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff3b0065430 0x7ff3b00678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff3c80fdea0 0x7ff3c81135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 --2- 172.21.15.153:0/2473453167 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff3c8065030 0x7ff3c8113070 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 >> 172.21.15.153:0/2473453167 conn(0x7ff3c8072110 msgr2=0x7ff3c8105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 shutdown_connections 2024-06-06T03:14:48.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:48.836+0000 7ff3d0627640 1 -- 172.21.15.153:0/2473453167 wait complete. 2024-06-06T03:14:49.102 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:49 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2473453167' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T03:14:49.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:49 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2473453167' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-06T03:14:49.537 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":19,"flags":0,"active_gid":14212,"active_name":"smithi153.gidtue","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":2373762556},{"type":"v1","addr":"172.21.15.153:6801","nonce":2373762556}]},"active_addr":"172.21.15.153:6801/2373762556","active_change":"2024-06-06T03:11:10.698726+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi173.zwtggy","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.153:8443/","prometheus":"http://172.21.15.153:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":3894205622}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":569370290}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2598405000}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.153:0","nonce":2389984450}]}]} 2024-06-06T03:14:49.541 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-06T03:14:49.541 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-06T03:14:49.541 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T03:14:49.792 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:50.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:50 smithi153 ceph-mon[29300]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:50.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:50 smithi173 ceph-mon[35650]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:51.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 -- 172.21.15.153:0/1500787647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 msgr2=0x7f2d180fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:51.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/1500787647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d180fc300 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f2d080048d0 tx=0x7f2d08009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 -- 172.21.15.153:0/1500787647 shutdown_connections 2024-06-06T03:14:51.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/1500787647 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d180fc300 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/1500787647 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 0x7f2d181026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 -- 172.21.15.153:0/1500787647 >> 172.21.15.153:0/1500787647 conn(0x7f2d180f7e80 msgr2=0x7f2d180fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 -- 172.21.15.153:0/1500787647 shutdown_connections 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.394+0000 7f2d1ec92640 1 -- 172.21.15.153:0/1500787647 wait complete. 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 Processor -- start 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 -- start start 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 0x7f2d18113090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d18113c60 con 0x7f2d18102c70 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ec92640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1810d010 con 0x7f2d181022d0 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56750/0 (socket says 172.21.15.153:56750) 2024-06-06T03:14:51.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 -- 172.21.15.153:0/3821950053 learned_addr learned my addr 172.21.15.153:0/3821950053 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d1ca2e640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 0x7f2d18113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 -- 172.21.15.153:0/3821950053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 msgr2=0x7f2d18113090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 0x7f2d18113090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.395+0000 7f2d0ffff640 1 -- 172.21.15.153:0/3821950053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d08005040 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d0ffff640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2d08003080 tx=0x7f2d08009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d08038070 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2d08008ee0 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1810d290 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2d0803d870 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.396+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d1810d750 con 0x7f2d18102c70 2024-06-06T03:14:51.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.397+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d180fda40 con 0x7f2d18102c70 2024-06-06T03:14:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.397+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2d08030020 con 0x7f2d18102c70 2024-06-06T03:14:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.398+0000 7f2d0d7fa640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 0x7f2d000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.398+0000 7f2d1ca2e640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 0x7f2d000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.398+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2d080a2180 con 0x7f2d18102c70 2024-06-06T03:14:51.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.398+0000 7f2d1ca2e640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 0x7f2d000678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009ac0 tx=0x7f2d14009120 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:51.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.402+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2d080706b0 con 0x7f2d18102c70 2024-06-06T03:14:51.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.569+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d18109490 con 0x7f2d18102c70 2024-06-06T03:14:51.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.570+0000 7f2d0d7fa640 1 -- 172.21.15.153:0/3821950053 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11131 (secure 0 0 0) 0x7f2d08073cd0 con 0x7f2d18102c70 2024-06-06T03:14:51.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:51.570 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","created":"2024-06-06T03:09:14.685479+0000","modified":"2024-06-06T03:14:31.988451+0000","last_up_change":"2024-06-06T03:14:30.983590+0000","last_in_change":"2024-06-06T03:14:12.198580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T03:13:21.002067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"faf72683-4c0d-4d8b-9f4a-51826f225303","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6803","nonce":2356787937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6805","nonce":2356787937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6809","nonce":2356787937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6807","nonce":2356787937}]},"public_addr":"172.21.15.153:6803/2356787937","cluster_addr":"172.21.15.153:6805/2356787937","heartbeat_back_addr":"172.21.15.153:6809/2356787937","heartbeat_front_addr":"172.21.15.153:6807/2356787937","state":["exists","up"]},{"osd":1,"uuid":"ad70997e-451a-4dfc-8b5a-2ce4a3851cb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6811","nonce":4085830578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6813","nonce":4085830578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6817","nonce":4085830578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6815","nonce":4085830578}]},"public_addr":"172.21.15.153:6811/4085830578","cluster_addr":"172.21.15.153:6813/4085830578","heartbeat_back_addr":"172.21.15.153:6817/4085830578","heartbeat_front_addr":"172.21.15.153:6815/4085830578","state":["exists","up"]},{"osd":2,"uuid":"4d203663-86cb-4662-a7dd-2353c0a50101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6819","nonce":858759998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6821","nonce":858759998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6825","nonce":858759998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6823","nonce":858759998}]},"public_addr":"172.21.15.153:6819/858759998","cluster_addr":"172.21.15.153:6821/858759998","heartbeat_back_addr":"172.21.15.153:6825/858759998","heartbeat_front_addr":"172.21.15.153:6823/858759998","state":["exists","up"]},{"osd":3,"uuid":"e1410b23-185c-457c-a009-e90924e8dbaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6801","nonce":1222772148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6803","nonce":1222772148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6807","nonce":1222772148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6805","nonce":1222772148}]},"public_addr":"172.21.15.173:6801/1222772148","cluster_addr":"172.21.15.173:6803/1222772148","heartbeat_back_addr":"172.21.15.173:6807/1222772148","heartbeat_front_addr":"172.21.15.173:6805/1222772148","state":["exists","up"]},{"osd":4,"uuid":"6d58503b-bf9e-48b6-a984-cf8476ce3f6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6809","nonce":2520823922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6811","nonce":2520823922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6815","nonce":2520823922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6813","nonce":2520823922}]},"public_addr":"172.21.15.173:6809/2520823922","cluster_addr":"172.21.15.173:6811/2520823922","heartbeat_back_addr":"172.21.15.173:6815/2520823922","heartbeat_front_addr":"172.21.15.173:6813/2520823922","state":["exists","up"]},{"osd":5,"uuid":"c32699eb-9044-467a-b18e-f627e5991fa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6817","nonce":2901238167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6819","nonce":2901238167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6823","nonce":2901238167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6821","nonce":2901238167}]},"public_addr":"172.21.15.173:6817/2901238167","cluster_addr":"172.21.15.173:6819/2901238167","heartbeat_back_addr":"172.21.15.173:6823/2901238167","heartbeat_front_addr":"172.21.15.173:6821/2901238167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:20.508700+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:49.716707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:17.772334+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:40.350532+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:04.456105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:29.082829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2262799474":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/372809069":"2024-06-07T03:11:10.698615+0000","172.21.15.153:6800/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/161244006":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6800/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6801/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/3807092861":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/295427621":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/1168792425":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6800/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/2789596196":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/3471374002":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/2811436119":"2024-06-07T03:11:10.698615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T03:14:51.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 msgr2=0x7f2d000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:51.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 0x7f2d000678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009ac0 tx=0x7f2d14009120 comp rx=0 tx=0).stop 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 msgr2=0x7f2d181135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2d08003080 tx=0x7f2d08009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 shutdown_connections 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d18102c70 0x7f2d181135d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f2d00065430 0x7f2d000678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 --2- 172.21.15.153:0/3821950053 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f2d181022d0 0x7f2d18113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 >> 172.21.15.153:0/3821950053 conn(0x7f2d180f7e80 msgr2=0x7f2d180f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 shutdown_connections 2024-06-06T03:14:51.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:51.572+0000 7f2d1ec92640 1 -- 172.21.15.153:0/3821950053 wait complete. 2024-06-06T03:14:52.109 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:52 smithi173 ceph-mon[35650]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:52.109 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:52 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3821950053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:52.269 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:52 smithi153 ceph-mon[29300]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:52.269 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:52 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3821950053' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:52.294 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-06T03:14:52.294 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-06-06T03:14:52.544 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:54.090 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:54 smithi153 ceph-mon[29300]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:54.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/1794243378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c78102370 msgr2=0x7f5c78102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/1794243378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c78102370 0x7f5c78102740 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c700048d0 tx=0x7f5c70009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/1794243378 shutdown_connections 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/1794243378 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c780fa9e0 0x7f5c780fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/1794243378 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c78102370 0x7f5c78102740 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/1794243378 >> 172.21.15.153:0/1794243378 conn(0x7f5c78070a40 msgr2=0x7f5c78070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/1794243378 shutdown_connections 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.264+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/1794243378 wait complete. 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 Processor -- start 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 -- start start 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 0x7f5c78113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:54.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c78113ce0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ef3d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7810d090 con 0x7f5c78102370 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ccd9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ccd9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56766/0 (socket says 172.21.15.153:56766) 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c7ccd9640 1 -- 172.21.15.153:0/373536684 learned_addr learned my addr 172.21.15.153:0/373536684 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.265+0000 7f5c77fff640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 0x7f5c78113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ccd9640 1 -- 172.21.15.153:0/373536684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 msgr2=0x7f5c78113650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ccd9640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 0x7f5c78113650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ccd9640 1 -- 172.21.15.153:0/373536684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70005040 con 0x7f5c780fa9e0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c77fff640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 0x7f5c78113650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ccd9640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c70003080 tx=0x7f5c70009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c70038070 con 0x7f5c780fa9e0 2024-06-06T03:14:54.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5c70008ee0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7810d310 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5c7003d870 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.266+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7810d7d0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.267+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c78065530 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.267+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5c70030020 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.268+0000 7f5c757fa640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 0x7f5c640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.268+0000 7f5c77fff640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 0x7f5c640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.268+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5c700a2180 con 0x7f5c780fa9e0 2024-06-06T03:14:54.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.268+0000 7f5c77fff640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 0x7f5c640678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c7810e9d0 tx=0x7f5c68009610 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:14:54.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.272+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5c700706b0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:54 smithi173 ceph-mon[35650]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:54.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.439+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c7804b4e0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.440+0000 7f5c757fa640 1 -- 172.21.15.153:0/373536684 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+11131 (secure 0 0 0) 0x7f5c70073cd0 con 0x7f5c780fa9e0 2024-06-06T03:14:54.440 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:14:54.440 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":40,"fsid":"0b80117e-23b2-11ef-bc9b-c7b262605968","created":"2024-06-06T03:09:14.685479+0000","modified":"2024-06-06T03:14:31.988451+0000","last_up_change":"2024-06-06T03:14:30.983590+0000","last_in_change":"2024-06-06T03:14:12.198580+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-06T03:13:21.002067+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"faf72683-4c0d-4d8b-9f4a-51826f225303","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6803","nonce":2356787937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6805","nonce":2356787937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6809","nonce":2356787937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":2356787937},{"type":"v1","addr":"172.21.15.153:6807","nonce":2356787937}]},"public_addr":"172.21.15.153:6803/2356787937","cluster_addr":"172.21.15.153:6805/2356787937","heartbeat_back_addr":"172.21.15.153:6809/2356787937","heartbeat_front_addr":"172.21.15.153:6807/2356787937","state":["exists","up"]},{"osd":1,"uuid":"ad70997e-451a-4dfc-8b5a-2ce4a3851cb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6811","nonce":4085830578}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6812","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6813","nonce":4085830578}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6816","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6817","nonce":4085830578}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6814","nonce":4085830578},{"type":"v1","addr":"172.21.15.153:6815","nonce":4085830578}]},"public_addr":"172.21.15.153:6811/4085830578","cluster_addr":"172.21.15.153:6813/4085830578","heartbeat_back_addr":"172.21.15.153:6817/4085830578","heartbeat_front_addr":"172.21.15.153:6815/4085830578","state":["exists","up"]},{"osd":2,"uuid":"4d203663-86cb-4662-a7dd-2353c0a50101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6818","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6819","nonce":858759998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6820","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6821","nonce":858759998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6824","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6825","nonce":858759998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6822","nonce":858759998},{"type":"v1","addr":"172.21.15.153:6823","nonce":858759998}]},"public_addr":"172.21.15.153:6819/858759998","cluster_addr":"172.21.15.153:6821/858759998","heartbeat_back_addr":"172.21.15.153:6825/858759998","heartbeat_front_addr":"172.21.15.153:6823/858759998","state":["exists","up"]},{"osd":3,"uuid":"e1410b23-185c-457c-a009-e90924e8dbaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6800","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6801","nonce":1222772148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6802","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6803","nonce":1222772148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6806","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6807","nonce":1222772148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6804","nonce":1222772148},{"type":"v1","addr":"172.21.15.173:6805","nonce":1222772148}]},"public_addr":"172.21.15.173:6801/1222772148","cluster_addr":"172.21.15.173:6803/1222772148","heartbeat_back_addr":"172.21.15.173:6807/1222772148","heartbeat_front_addr":"172.21.15.173:6805/1222772148","state":["exists","up"]},{"osd":4,"uuid":"6d58503b-bf9e-48b6-a984-cf8476ce3f6d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6808","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6809","nonce":2520823922}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6810","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6811","nonce":2520823922}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6814","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6815","nonce":2520823922}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6812","nonce":2520823922},{"type":"v1","addr":"172.21.15.173:6813","nonce":2520823922}]},"public_addr":"172.21.15.173:6809/2520823922","cluster_addr":"172.21.15.173:6811/2520823922","heartbeat_back_addr":"172.21.15.173:6815/2520823922","heartbeat_front_addr":"172.21.15.173:6813/2520823922","state":["exists","up"]},{"osd":5,"uuid":"c32699eb-9044-467a-b18e-f627e5991fa8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6816","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6817","nonce":2901238167}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6818","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6819","nonce":2901238167}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6822","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6823","nonce":2901238167}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.173:6820","nonce":2901238167},{"type":"v1","addr":"172.21.15.173:6821","nonce":2901238167}]},"public_addr":"172.21.15.173:6817/2901238167","cluster_addr":"172.21.15.173:6819/2901238167","heartbeat_back_addr":"172.21.15.173:6823/2901238167","heartbeat_front_addr":"172.21.15.173:6821/2901238167","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:20.508700+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:12:49.716707+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:17.772334+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:13:40.350532+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:04.456105+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-06T03:14:29.082829+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.153:0/2262799474":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/372809069":"2024-06-07T03:11:10.698615+0000","172.21.15.153:6800/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/161244006":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6800/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/3563458677":"2024-06-07T03:09:35.094463+0000","172.21.15.153:6801/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/3807092861":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/295427621":"2024-06-07T03:09:35.094463+0000","172.21.15.153:0/1168792425":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6800/659590756":"2024-06-07T03:11:10.698615+0000","172.21.15.153:0/2789596196":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/3471374002":"2024-06-07T03:10:10.221154+0000","172.21.15.153:6801/1921588694":"2024-06-07T03:10:10.221154+0000","172.21.15.153:0/2811436119":"2024-06-07T03:11:10.698615+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 msgr2=0x7f5c640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 0x7f5c640678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5c7810e9d0 tx=0x7f5c68009610 comp rx=0 tx=0).stop 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 msgr2=0x7f5c78113110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5c70003080 tx=0x7f5c70009da0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 shutdown_connections 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5c64065430 0x7f5c640678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5c78102370 0x7f5c78113650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 --2- 172.21.15.153:0/373536684 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5c780fa9e0 0x7f5c78113110 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 >> 172.21.15.153:0/373536684 conn(0x7f5c78070a40 msgr2=0x7f5c780f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:14:54.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 shutdown_connections 2024-06-06T03:14:54.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:14:54.442+0000 7f5c7ef3d640 1 -- 172.21.15.153:0/373536684 wait complete. 2024-06-06T03:14:55.208 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:55 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/373536684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:55.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-06T03:14:55.235 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-06T03:14:55.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-06T03:14:55.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-06T03:14:55.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-06T03:14:55.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-06T03:14:55.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:55 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/373536684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-06T03:14:55.530 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:55.533 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:56 smithi153 ceph-mon[29300]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:56.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:56 smithi173 ceph-mon[35650]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:56.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:14:56.805 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:56.805 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:58.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:14:58 smithi153 ceph-mon[29300]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:58.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:14:58 smithi173 ceph-mon[35650]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:14:59.750 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:14:59.751 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:00.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:00 smithi173 ceph-mon[35650]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:00.373 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:00 smithi153 ceph-mon[29300]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/394612162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c541022c0 msgr2=0x7f3c54102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/394612162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c541022c0 0x7f3c54102690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3c480048d0 tx=0x7f3c48009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/394612162 shutdown_connections 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/394612162 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c54102c60 0x7f3c540fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/394612162 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c541022c0 0x7f3c54102690 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/394612162 >> 172.21.15.153:0/394612162 conn(0x7f3c540f7e50 msgr2=0x7f3c540fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/394612162 shutdown_connections 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.946+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/394612162 wait complete. 2024-06-06T03:15:00.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 Processor -- start 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 -- start start 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c541022c0 0x7f3c54113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c54113e70 con 0x7f3c54102c60 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c5b0fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5410d2b0 con 0x7f3c541022c0 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55970/0 (socket says 172.21.15.153:55970) 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c53fff640 1 -- 172.21.15.153:0/2120037646 learned_addr learned my addr 172.21.15.153:0/2120037646 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.947+0000 7f3c58e99640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c541022c0 0x7f3c54113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c53fff640 1 -- 172.21.15.153:0/2120037646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c541022c0 msgr2=0x7f3c54113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c53fff640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c541022c0 0x7f3c54113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c53fff640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c48005040 con 0x7f3c54102c60 2024-06-06T03:15:00.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c53fff640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c830 tx=0x7f3c40004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4000a850 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5410d590 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c4000ae80 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c40012ea0 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.948+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5410db30 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.949+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3c540fda30 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.949+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3c4000a9b0 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c517fa640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 0x7f3c440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3c40082700 con 0x7f3c54102c60 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c58e99640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 0x7f3c440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c517fa640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 0x7f3c4406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:00.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3c4406dc70 con 0x7f3c4406b1b0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c40084d50 con 0x7f3c54102c60 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c5969a640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 0x7f3c4406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c58e99640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 0x7f3c440678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c48000c00 tx=0x7f3c48035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.950+0000 7f3c5969a640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 0x7f3c4406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.951+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== osd.2 v2:172.21.15.153:6818/858759998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f3c4406dc70 con 0x7f3c4406b1b0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.964+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 --> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3c5410e070 con 0x7f3c4406b1b0 2024-06-06T03:15:00.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.964+0000 7f3c517fa640 1 -- 172.21.15.153:0/2120037646 <== osd.2 v2:172.21.15.153:6818/858759998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3c5410e070 con 0x7f3c4406b1b0 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.964+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 msgr2=0x7f3c4406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.964+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 0x7f3c4406d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 msgr2=0x7f3c440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 0x7f3c440678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3c48000c00 tx=0x7f3c48035040 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 msgr2=0x7f3c54113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3c4000c830 tx=0x7f3c40004430 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 shutdown_connections 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6818/858759998,v1:172.21.15.153:6819/858759998] conn(0x7f3c4406b1b0 0x7f3c4406d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c54102c60 0x7f3c54113870 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c44065430 0x7f3c440678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 --2- 172.21.15.153:0/2120037646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c541022c0 0x7f3c54113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 >> 172.21.15.153:0/2120037646 conn(0x7f3c540f7e50 msgr2=0x7f3c540f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:00.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 shutdown_connections 2024-06-06T03:15:00.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:00.965+0000 7f3c5b0fd640 1 -- 172.21.15.153:0/2120037646 wait complete. 2024-06-06T03:15:01.656 INFO:teuthology.orchestra.run.smithi153.stdout:81604378648 2024-06-06T03:15:01.656 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.882+0000 7fd1581e4640 1 -- 172.21.15.153:0/14979625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fbe80 msgr2=0x7fd1500fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.882+0000 7fd1581e4640 1 --2- 172.21.15.153:0/14979625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fbe80 0x7fd1500fcd80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd1400048d0 tx=0x7fd140009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 -- 172.21.15.153:0/14979625 shutdown_connections 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 --2- 172.21.15.153:0/14979625 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fbe80 0x7fd1500fcd80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 --2- 172.21.15.153:0/14979625 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fb570 0x7fd1500fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 -- 172.21.15.153:0/14979625 >> 172.21.15.153:0/14979625 conn(0x7fd150072110 msgr2=0x7fd150072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 -- 172.21.15.153:0/14979625 shutdown_connections 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 -- 172.21.15.153:0/14979625 wait complete. 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 Processor -- start 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 -- start start 2024-06-06T03:15:01.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.883+0000 7fd1581e4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 0x7fd150119af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1581e4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15011a140 con 0x7fd1500fb570 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1581e4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd150113550 con 0x7fd1500fbe80 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55978/0 (socket says 172.21.15.153:55978) 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 -- 172.21.15.153:0/3953203524 learned_addr learned my addr 172.21.15.153:0/3953203524 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd15577f640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 0x7fd150119af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 -- 172.21.15.153:0/3953203524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 msgr2=0x7fd150119af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 0x7fd150119af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd140005040 con 0x7fd1500fb570 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd15577f640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 0x7fd150119af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd155f80640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c830 tx=0x7fd14c004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:01.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14c00a850 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd14c00ae80 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd14c012ea0 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd150113830 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.884+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd150113dd0 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.885+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd1501143a0 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.886+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd14c012400 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd1467fc640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 0x7fd13c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd14c082620 con 0x7fd1500fb570 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd15577f640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 0x7fd13c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd1467fc640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 0x7fd13c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd13c06dc70 con 0x7fd13c06b1b0 2024-06-06T03:15:01.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd14c084c90 con 0x7fd1500fb570 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd156781640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 0x7fd13c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd15577f640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 0x7fd13c0678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd140009f80 tx=0x7fd140009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.887+0000 7fd156781640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 0x7fd13c06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.888+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== osd.1 v2:172.21.15.153:6810/4085830578 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd13c06dc70 con 0x7fd13c06b1b0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.901+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 --> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd1500fc920 con 0x7fd13c06b1b0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.901+0000 7fd1467fc640 1 -- 172.21.15.153:0/3953203524 <== osd.1 v2:172.21.15.153:6810/4085830578 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd1500fc920 con 0x7fd13c06b1b0 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.901+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 msgr2=0x7fd13c06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.901+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 0x7fd13c06d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 msgr2=0x7fd13c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 0x7fd13c0678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd140009f80 tx=0x7fd140009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 msgr2=0x7fd1501195b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00c830 tx=0x7fd14c004430 comp rx=0 tx=0).stop 2024-06-06T03:15:01.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 shutdown_connections 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6810/4085830578,v1:172.21.15.153:6811/4085830578] conn(0x7fd13c06b1b0 0x7fd13c06d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd13c065430 0x7fd13c0678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd1500fbe80 0x7fd150119af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 --2- 172.21.15.153:0/3953203524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd1500fb570 0x7fd1501195b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 >> 172.21.15.153:0/3953203524 conn(0x7fd150072110 msgr2=0x7fd150109400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 shutdown_connections 2024-06-06T03:15:01.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:01.902+0000 7fd1581e4640 1 -- 172.21.15.153:0/3953203524 wait complete. 2024-06-06T03:15:02.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:02 smithi153 ceph-mon[29300]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:02.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:02 smithi173 ceph-mon[35650]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:02.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 -- 172.21.15.153:0/796426208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 msgr2=0x7f9c600fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/796426208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c600fb940 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9c540048d0 tx=0x7f9c54009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 -- 172.21.15.153:0/796426208 shutdown_connections 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/796426208 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 0x7f9c600fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/796426208 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c600fb940 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 -- 172.21.15.153:0/796426208 >> 172.21.15.153:0/796426208 conn(0x7f9c60072110 msgr2=0x7f9c60072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 -- 172.21.15.153:0/796426208 shutdown_connections 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.761+0000 7f9c66dd1640 1 -- 172.21.15.153:0/796426208 wait complete. 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 Processor -- start 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 -- start start 2024-06-06T03:15:02.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 0x7f9c60112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c60112a60 con 0x7f9c600fb570 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c66dd1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6010bea0 con 0x7f9c600fbe80 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c64b6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c64b6d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56012/0 (socket says 172.21.15.153:56012) 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c64b6d640 1 -- 172.21.15.153:0/1455004705 learned_addr learned my addr 172.21.15.153:0/1455004705 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.762+0000 7f9c5ffff640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 0x7f9c60112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c64b6d640 1 -- 172.21.15.153:0/1455004705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 msgr2=0x7f9c60112460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:02.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c64b6d640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 0x7f9c60112460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c64b6d640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54005040 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c64b6d640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9c54003080 tx=0x7f9c54009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c54038070 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6010c120 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c54008ee0 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c5403d870 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.763+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6010c5e0 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.764+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9c6010cac0 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9c54030020 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 0x7f9c500678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c540a24b0 con 0x7f9c600fb570 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5ffff640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 0x7f9c500678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:02.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 0x7f9c5006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9c5006dc70 con 0x7f9c5006b1b0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c540a2890 con 0x7f9c600fb570 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c6536e640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 0x7f9c5006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.765+0000 7f9c5ffff640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 0x7f9c500678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9c6010d7e0 tx=0x7f9c4c009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.766+0000 7f9c6536e640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 0x7f9c5006d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.766+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== osd.0 v2:172.21.15.153:6802/2356787937 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f9c5006dc70 con 0x7f9c5006b1b0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 --> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9c600fc920 con 0x7f9c5006b1b0 2024-06-06T03:15:02.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c5d7fa640 1 -- 172.21.15.153:0/1455004705 <== osd.0 v2:172.21.15.153:6802/2356787937 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9c600fc920 con 0x7f9c5006b1b0 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 msgr2=0x7f9c5006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 0x7f9c5006d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 msgr2=0x7f9c500678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 0x7f9c500678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9c6010d7e0 tx=0x7f9c4c009040 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 msgr2=0x7f9c60111f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9c54003080 tx=0x7f9c54009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 shutdown_connections 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6802/2356787937,v1:172.21.15.153:6803/2356787937] conn(0x7f9c5006b1b0 0x7f9c5006d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c50065430 0x7f9c500678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c600fbe80 0x7f9c60112460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 --2- 172.21.15.153:0/1455004705 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c600fb570 0x7f9c60111f20 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:02.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 >> 172.21.15.153:0/1455004705 conn(0x7f9c60072110 msgr2=0x7f9c600f86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:02.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 shutdown_connections 2024-06-06T03:15:02.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.780+0000 7f9c66dd1640 1 -- 172.21.15.153:0/1455004705 wait complete. 2024-06-06T03:15:03.838 INFO:teuthology.orchestra.run.smithi153.stdout:38654705697 2024-06-06T03:15:03.838 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-06T03:15:03.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 -- 172.21.15.153:0/3922053837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 msgr2=0x7f42b00fb190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3922053837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b00fb190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f42a00048d0 tx=0x7f42a0009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 -- 172.21.15.153:0/3922053837 shutdown_connections 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3922053837 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b00fb190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3922053837 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 0x7f42b0101530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 -- 172.21.15.153:0/3922053837 >> 172.21.15.153:0/3922053837 conn(0x7f42b00f6d50 msgr2=0x7f42b00f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 -- 172.21.15.153:0/3922053837 shutdown_connections 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.661+0000 7f42b77e3640 1 -- 172.21.15.153:0/3922053837 wait complete. 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 Processor -- start 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 -- start start 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 0x7f42b0111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:03.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b0112b30 con 0x7f42b0101b00 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b77e3640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b010bee0 con 0x7f42b0101160 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b4d7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b4d7e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55994/0 (socket says 172.21.15.153:55994) 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.662+0000 7f42b4d7e640 1 -- 172.21.15.153:0/3948302233 learned_addr learned my addr 172.21.15.153:0/3948302233 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b557f640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 0x7f42b0111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b4d7e640 1 -- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 msgr2=0x7f42b0111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b4d7e640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 0x7f42b0111f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b4d7e640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a0005040 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b4d7e640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f42a0003080 tx=0x7f42a0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42a0038070 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f42a0008ee0 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b010c160 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f42a003d870 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.663+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b010c620 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.664+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f42b010cbc0 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.664+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f42a0030020 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42a5ffb640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 0x7f429c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f42a00a22e0 con 0x7f42b0101b00 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42b557f640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 0x7f429c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:03.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42a5ffb640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 0x7f429c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f429c06dc70 con 0x7f429c06b1b0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42a00a4890 con 0x7f42b0101b00 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42b557f640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 0x7f429c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42ac006fd0 tx=0x7f42ac008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.665+0000 7f42b5d80640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 0x7f429c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.666+0000 7f42b5d80640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 0x7f429c06d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.667+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== osd.3 v2:172.21.15.173:6800/1222772148 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f429c06dc70 con 0x7f429c06b1b0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.679+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 --> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f42b00f8620 con 0x7f429c06b1b0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42a5ffb640 1 -- 172.21.15.153:0/3948302233 <== osd.3 v2:172.21.15.173:6800/1222772148 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f42b00f8620 con 0x7f429c06b1b0 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 msgr2=0x7f429c06d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 0x7f429c06d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 msgr2=0x7f429c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:03.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 0x7f429c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f42ac006fd0 tx=0x7f42ac008040 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 msgr2=0x7f42b01124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f42a0003080 tx=0x7f42a0009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 shutdown_connections 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:6800/1222772148,v1:172.21.15.173:6801/1222772148] conn(0x7f429c06b1b0 0x7f429c06d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f42b0101b00 0x7f42b01124a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f429c065430 0x7f429c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 --2- 172.21.15.153:0/3948302233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f42b0101160 0x7f42b0111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.680+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 >> 172.21.15.153:0/3948302233 conn(0x7f42b00f6d50 msgr2=0x7f42b00f7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.681+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 shutdown_connections 2024-06-06T03:15:03.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:02.681+0000 7f42b77e3640 1 -- 172.21.15.153:0/3948302233 wait complete. 2024-06-06T03:15:04.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:04 smithi173 ceph-mon[35650]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:04.475 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:04 smithi153 ceph-mon[29300]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:04.499 INFO:teuthology.orchestra.run.smithi153.stdout:60129542173 2024-06-06T03:15:04.500 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-06T03:15:05.723 INFO:teuthology.orchestra.run.smithi153.stdout:111669149715 2024-06-06T03:15:05.723 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-06T03:15:06.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:06 smithi173 ceph-mon[35650]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:06.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:06 smithi153 ceph-mon[29300]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:07.278 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 -- 172.21.15.153:0/2376776454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 msgr2=0x7fdf780fd950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2376776454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf780fd950 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0048d0 tx=0x7fdf6c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 -- 172.21.15.153:0/2376776454 shutdown_connections 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2376776454 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf781063c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2376776454 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf780fd950 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 -- 172.21.15.153:0/2376776454 >> 172.21.15.153:0/2376776454 conn(0x7fdf780f6d90 msgr2=0x7fdf780f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:07.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 -- 172.21.15.153:0/2376776454 shutdown_connections 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.521+0000 7fdf800e5640 1 -- 172.21.15.153:0/2376776454 wait complete. 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 Processor -- start 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 -- start start 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf7810e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7810af80 con 0x7fdf780fd580 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf800e5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7810b0c0 con 0x7fdf780fde90 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56028/0 (socket says 172.21.15.153:56028) 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 -- 172.21.15.153:0/2096284939 learned_addr learned my addr 172.21.15.153:0/2096284939 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7d680640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf7810e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:07.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 -- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 msgr2=0x7fdf7810e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:07.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf7810e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c005040 con 0x7fdf780fd580 2024-06-06T03:15:07.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7d680640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf7810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:15:07.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.522+0000 7fdf7de81640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0043d0 tx=0x7fdf6c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:07.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6c038070 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdf6c032970 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7810b340 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdf6c03da60 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7810b800 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.523+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdf780fa9f0 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.524+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdf6c030020 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.524+0000 7fdf6a7fc640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 0x7fdf640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.524+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdf6c0a25a0 con 0x7fdf780fd580 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf7d680640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 0x7fdf640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf6a7fc640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 0x7fdf6406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:07.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdf6406dc70 con 0x7fdf6406b1b0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf6c0a4bf0 con 0x7fdf780fd580 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf7e682640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 0x7fdf6406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf7d680640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 0x7fdf640678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7810c870 tx=0x7fdf74009f30 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.525+0000 7fdf7e682640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 0x7fdf6406d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.526+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== osd.4 v2:172.21.15.173:6808/2520823922 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fdf6406dc70 con 0x7fdf6406b1b0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.539+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 --> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdf78109580 con 0x7fdf6406b1b0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf6a7fc640 1 -- 172.21.15.153:0/2096284939 <== osd.4 v2:172.21.15.173:6808/2520823922 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fdf78109580 con 0x7fdf6406b1b0 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 msgr2=0x7fdf6406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 0x7fdf6406d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 msgr2=0x7fdf640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 0x7fdf640678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fdf7810c870 tx=0x7fdf74009f30 comp rx=0 tx=0).stop 2024-06-06T03:15:07.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 msgr2=0x7fdf7810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0043d0 tx=0x7fdf6c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 shutdown_connections 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:6808/2520823922,v1:172.21.15.173:6809/2520823922] conn(0x7fdf6406b1b0 0x7fdf6406d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdf64065430 0x7fdf640678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdf780fde90 0x7fdf7810e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 --2- 172.21.15.153:0/2096284939 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdf780fd580 0x7fdf7810df80 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 >> 172.21.15.153:0/2096284939 conn(0x7fdf780f6d90 msgr2=0x7fdf780f8340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 shutdown_connections 2024-06-06T03:15:07.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:07.540+0000 7fdf800e5640 1 -- 172.21.15.153:0/2096284939 wait complete. 2024-06-06T03:15:07.546 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:07.547 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:08.170 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:08 smithi153 ceph-mon[29300]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:08.215 INFO:teuthology.orchestra.run.smithi153.stdout:146028888080 2024-06-06T03:15:08.216 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-06T03:15:08.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:08 smithi173 ceph-mon[35650]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:08.481 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/4224679024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 msgr2=0x7fc2a00fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/4224679024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a00fb1e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0048d0 tx=0x7fc28c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/4224679024 shutdown_connections 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/4224679024 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a00fb1e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/4224679024 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a01011b0 0x7fc2a0101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/4224679024 >> 172.21.15.153:0/4224679024 conn(0x7fc2a00f6d80 msgr2=0x7fc2a00f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/4224679024 shutdown_connections 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/4224679024 wait complete. 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.130+0000 7fc2a6b7b640 1 Processor -- start 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc2a6b7b640 1 -- start start 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc2a6b7b640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a01011b0 0x7fc2a0111f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc2a6b7b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc2a6b7b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0112a50 con 0x7fc2a0101b50 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc2a6b7b640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a010be90 con 0x7fc2a01011b0 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:10.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36110/0 (socket says 172.21.15.153:36110) 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 -- 172.21.15.153:0/3888837889 learned_addr learned my addr 172.21.15.153:0/3888837889 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 -- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a01011b0 msgr2=0x7fc2a0111f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a01011b0 0x7fc2a0111f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c005040 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29ffff640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0043d0 tx=0x7fc28c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.131+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c038070 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.132+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc28c02abd0 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.132+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a010c110 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.132+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc28c0336f0 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.132+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a010c650 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.132+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc2a010cbf0 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.133+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc28c030020 con 0x7fc2a0101b50 2024-06-06T03:15:10.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.133+0000 7fc29d7fa640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 0x7fc2900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.133+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc28c0a2290 con 0x7fc2a0101b50 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.133+0000 7fc2a4917640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 0x7fc2900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc29d7fa640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 0x7fc29006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc29006dc70 con 0x7fc29006b1b0 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc28c0a48e0 con 0x7fc2a0101b50 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc2a5118640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 0x7fc29006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc2a4917640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 0x7fc2900678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc294004c80 tx=0x7fc294009c50 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:10.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.134+0000 7fc2a5118640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 0x7fc29006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.135+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== osd.5 v2:172.21.15.173:6816/2901238167 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc29006dc70 con 0x7fc29006b1b0 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.148+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 --> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc2a00f86b0 con 0x7fc29006b1b0 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc29d7fa640 1 -- 172.21.15.153:0/3888837889 <== osd.5 v2:172.21.15.173:6816/2901238167 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc2a00f86b0 con 0x7fc29006b1b0 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 msgr2=0x7fc29006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 0x7fc29006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 msgr2=0x7fc2900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 0x7fc2900678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc294004c80 tx=0x7fc294009c50 comp rx=0 tx=0).stop 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 msgr2=0x7fc2a0112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0043d0 tx=0x7fc28c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 shutdown_connections 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:6816/2901238167,v1:172.21.15.173:6817/2901238167] conn(0x7fc29006b1b0 0x7fc29006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2a0101b50 0x7fc2a0112450 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fc290065430 0x7fc2900678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 --2- 172.21.15.153:0/3888837889 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fc2a01011b0 0x7fc2a0111f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 >> 172.21.15.153:0/3888837889 conn(0x7fc2a00f6d80 msgr2=0x7fc2a00f7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 shutdown_connections 2024-06-06T03:15:10.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:10.149+0000 7fc2a6b7b640 1 -- 172.21.15.153:0/3888837889 wait complete. 2024-06-06T03:15:10.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:10 smithi153 ceph-mon[29300]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:10.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:10 smithi173 ceph-mon[35650]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:11.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:11.349 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:12.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:12 smithi173 ceph-mon[35650]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:12.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:12 smithi153 ceph-mon[29300]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:13.160 INFO:teuthology.orchestra.run.smithi153.stdout:167503724554 2024-06-06T03:15:13.161 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-06T03:15:13.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 -- 172.21.15.153:0/18122953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 msgr2=0x7f19340fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:13.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 --2- 172.21.15.153:0/18122953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f19340fc2d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f19280048d0 tx=0x7f1928009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 -- 172.21.15.153:0/18122953 shutdown_connections 2024-06-06T03:15:13.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 --2- 172.21.15.153:0/18122953 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f19340fc2d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 --2- 172.21.15.153:0/18122953 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19341022a0 0x7f1934102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 -- 172.21.15.153:0/18122953 >> 172.21.15.153:0/18122953 conn(0x7f19340f7e70 msgr2=0x7f19340fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 -- 172.21.15.153:0/18122953 shutdown_connections 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.736+0000 7f193a5d7640 1 -- 172.21.15.153:0/18122953 wait complete. 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 Processor -- start 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 -- start start 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19341022a0 0x7f193406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1934070280 con 0x7f1934102c40 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f193a5d7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19340703f0 con 0x7f19341022a0 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f19337fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:13.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f19337fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36128/0 (socket says 172.21.15.153:36128) 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.737+0000 7f19337fe640 1 -- 172.21.15.153:0/4214788493 learned_addr learned my addr 172.21.15.153:0/4214788493 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f19337fe640 1 -- 172.21.15.153:0/4214788493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19341022a0 msgr2=0x7f193406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f19337fe640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19341022a0 0x7f193406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f19337fe640 1 -- 172.21.15.153:0/4214788493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1928005040 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f19337fe640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1928003080 tx=0x7f1928009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1928038070 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19341193e0 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1928008ee0 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f192803d870 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.738+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1934119760 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.739+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19340fda10 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.739+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1928030020 con 0x7f1934102c40 2024-06-06T03:15:13.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.740+0000 7f1930ff9640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 0x7f191c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:13.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.740+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f19280a2240 con 0x7f1934102c40 2024-06-06T03:15:13.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.740+0000 7f1933fff640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 0x7f191c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:13.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.740+0000 7f1933fff640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 0x7f191c0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19240097e0 tx=0x7f1924006d10 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:13.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.744+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1928070770 con 0x7f1934102c40 2024-06-06T03:15:13.762 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.922+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f193404b480 con 0x7f1934102c40 2024-06-06T03:15:13.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.922+0000 7f1930ff9640 1 -- 172.21.15.153:0/4214788493 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1928073d90 con 0x7f1934102c40 2024-06-06T03:15:13.923 INFO:teuthology.orchestra.run.smithi153.stdout:111669149717 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 msgr2=0x7f191c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 0x7f191c0678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f19240097e0 tx=0x7f1924006d10 comp rx=0 tx=0).stop 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 msgr2=0x7f193406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f1928003080 tx=0x7f1928009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 shutdown_connections 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1934102c40 0x7f193406fcb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f191c065430 0x7f191c0678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 --2- 172.21.15.153:0/4214788493 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f19341022a0 0x7f193406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:13.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.924+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 >> 172.21.15.153:0/4214788493 conn(0x7f19340f7e70 msgr2=0x7f19340f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.925+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 shutdown_connections 2024-06-06T03:15:13.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:13.925+0000 7f193a5d7640 1 -- 172.21.15.153:0/4214788493 wait complete. 2024-06-06T03:15:14.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:14 smithi153 ceph-mon[29300]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:14.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:14 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4214788493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T03:15:14.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:14 smithi173 ceph-mon[35650]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:14.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:14 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4214788493' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-06T03:15:14.648 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149715 got 111669149717 for osd.3 2024-06-06T03:15:14.648 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:15.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 -- 172.21.15.153:0/2518196093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 msgr2=0x7f7c100fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:15.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 --2- 172.21.15.153:0/2518196093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fa9e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7c040048d0 tx=0x7f7c04009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 -- 172.21.15.153:0/2518196093 shutdown_connections 2024-06-06T03:15:15.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 --2- 172.21.15.153:0/2518196093 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fa9e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 --2- 172.21.15.153:0/2518196093 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 0x7f7c101026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 -- 172.21.15.153:0/2518196093 >> 172.21.15.153:0/2518196093 conn(0x7f7c10070a40 msgr2=0x7f7c10070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:15.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 -- 172.21.15.153:0/2518196093 shutdown_connections 2024-06-06T03:15:15.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.460+0000 7f7c163f5640 1 -- 172.21.15.153:0/2518196093 wait complete. 2024-06-06T03:15:15.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 Processor -- start 2024-06-06T03:15:15.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 -- start start 2024-06-06T03:15:15.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 0x7f7c100fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100fc640 con 0x7f7c10102c90 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c163f5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c100fc7b0 con 0x7f7c101022f0 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c0f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c0f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36134/0 (socket says 172.21.15.153:36134) 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c0f7fe640 1 -- 172.21.15.153:0/4010337726 learned_addr learned my addr 172.21.15.153:0/4010337726 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.461+0000 7f7c0ffff640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 0x7f7c100fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0f7fe640 1 -- 172.21.15.153:0/4010337726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 msgr2=0x7f7c100fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0f7fe640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 0x7f7c100fbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0f7fe640 1 -- 172.21.15.153:0/4010337726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c04005040 con 0x7f7c10102c90 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0f7fe640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7c04003080 tx=0x7f7c04009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:15.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c04038070 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7c04008ee0 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c100fc9e0 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c04033760 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.462+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c10119680 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.463+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c10065510 con 0x7f7c10102c90 2024-06-06T03:15:15.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.464+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7c04030020 con 0x7f7c10102c90 2024-06-06T03:15:15.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.464+0000 7f7c0cff9640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 0x7f7bf80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:15.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.464+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7c040a1f90 con 0x7f7c10102c90 2024-06-06T03:15:15.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.464+0000 7f7c0ffff640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 0x7f7bf80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:15.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.465+0000 7f7c0ffff640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 0x7f7bf80678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c10065260 tx=0x7f7c00009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:15.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.468+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c040704c0 con 0x7f7c10102c90 2024-06-06T03:15:15.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.640+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7c101094e0 con 0x7f7c10102c90 2024-06-06T03:15:15.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.640+0000 7f7c0cff9640 1 -- 172.21.15.153:0/4010337726 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c04073ae0 con 0x7f7c10102c90 2024-06-06T03:15:15.640 INFO:teuthology.orchestra.run.smithi153.stdout:60129542176 2024-06-06T03:15:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 msgr2=0x7f7bf80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 0x7f7bf80678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7c10065260 tx=0x7f7c00009040 comp rx=0 tx=0).stop 2024-06-06T03:15:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 msgr2=0x7f7c100fc100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7c04003080 tx=0x7f7c04009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 shutdown_connections 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7c10102c90 0x7f7c100fc100 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7bf8065430 0x7f7bf80678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 --2- 172.21.15.153:0/4010337726 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7c101022f0 0x7f7c100fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 >> 172.21.15.153:0/4010337726 conn(0x7f7c10070a40 msgr2=0x7f7c10105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 shutdown_connections 2024-06-06T03:15:15.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:15.643+0000 7f7c163f5640 1 -- 172.21.15.153:0/4010337726 wait complete. 2024-06-06T03:15:16.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:16 smithi153 ceph-mon[29300]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:16.265 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:16 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4010337726' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T03:15:16.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:16 smithi173 ceph-mon[35650]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:16.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:16 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4010337726' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-06T03:15:16.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 -- 172.21.15.153:0/1789365888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 msgr2=0x7fb8e006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1789365888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e006f810 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc0048d0 tx=0x7fb8dc009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 -- 172.21.15.153:0/1789365888 shutdown_connections 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1789365888 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 0x7fb8e00701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1789365888 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e006f810 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 -- 172.21.15.153:0/1789365888 >> 172.21.15.153:0/1789365888 conn(0x7fb8e00f7f00 msgr2=0x7fb8e00fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 -- 172.21.15.153:0/1789365888 shutdown_connections 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 -- 172.21.15.153:0/1789365888 wait complete. 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.647+0000 7fb8e8090640 1 Processor -- start 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e8090640 1 -- start start 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e8090640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 0x7fb8e0119670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:16.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e8090640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e8090640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e011a220 con 0x7fb8e0071440 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e8090640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e0113610 con 0x7fb8e006fd50 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36150/0 (socket says 172.21.15.153:36150) 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 -- 172.21.15.153:0/1179001014 learned_addr learned my addr 172.21.15.153:0/1179001014 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e5e2c640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 0x7fb8e0119670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 -- 172.21.15.153:0/1179001014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 msgr2=0x7fb8e0119670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 0x7fb8e0119670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 -- 172.21.15.153:0/1179001014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8dc005040 con 0x7fb8e0071440 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.648+0000 7fb8e562b640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000c810 tx=0x7fb8d00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.649+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d000a7a0 con 0x7fb8e0071440 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.649+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8d000add0 con 0x7fb8e0071440 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.649+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e01138f0 con 0x7fb8e0071440 2024-06-06T03:15:16.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.649+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8d00115c0 con 0x7fb8e0071440 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.649+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e0113de0 con 0x7fb8e0071440 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.650+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e0070f50 con 0x7fb8e0071440 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.650+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb8d0012420 con 0x7fb8e0071440 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.650+0000 7fb8d67fc640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 0x7fb8cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.650+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb8d00828b0 con 0x7fb8e0071440 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.650+0000 7fb8e5e2c640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 0x7fb8cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.651+0000 7fb8e5e2c640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 0x7fb8cc0678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc000c00 tx=0x7fb8dc035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:16.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.654+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb8d0050de0 con 0x7fb8e0071440 2024-06-06T03:15:16.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.822+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb8e00fbb00 con 0x7fb8e0071440 2024-06-06T03:15:16.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.822+0000 7fb8d67fc640 1 -- 172.21.15.153:0/1179001014 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb8d0054400 con 0x7fb8e0071440 2024-06-06T03:15:16.822 INFO:teuthology.orchestra.run.smithi153.stdout:81604378650 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 msgr2=0x7fb8cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 0x7fb8cc0678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb8dc000c00 tx=0x7fb8dc035040 comp rx=0 tx=0).stop 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 msgr2=0x7fb8e0119bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb8d000c810 tx=0x7fb8d00043c0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 shutdown_connections 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8e0071440 0x7fb8e0119bd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8cc065430 0x7fb8cc0678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 --2- 172.21.15.153:0/1179001014 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8e006fd50 0x7fb8e0119670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 >> 172.21.15.153:0/1179001014 conn(0x7fb8e00f7f00 msgr2=0x7fb8e00f9080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:16.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 shutdown_connections 2024-06-06T03:15:16.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:16.824+0000 7fb8e8090640 1 -- 172.21.15.153:0/1179001014 wait complete. 2024-06-06T03:15:17.326 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:17 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1179001014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T03:15:17.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:17 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1179001014' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-06T03:15:18.021 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-06-06T03:15:18.022 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:18.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 -- 172.21.15.153:0/451174522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 msgr2=0x7f9cfc0fd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/451174522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc0fd540 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0048d0 tx=0x7f9cec009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 -- 172.21.15.153:0/451174522 shutdown_connections 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/451174522 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 0x7f9cfc0faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/451174522 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc0fd540 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 -- 172.21.15.153:0/451174522 >> 172.21.15.153:0/451174522 conn(0x7f9cfc070ab0 msgr2=0x7f9cfc070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 -- 172.21.15.153:0/451174522 shutdown_connections 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.204+0000 7f9d00a1e640 1 -- 172.21.15.153:0/451174522 wait complete. 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 Processor -- start 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 -- start start 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 0x7f9cfc115250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc115d90 con 0x7f9cfc0fd170 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9d00a1e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc10f1d0 con 0x7f9cfc0faa50 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36176/0 (socket says 172.21.15.153:36176) 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 -- 172.21.15.153:0/3282145662 learned_addr learned my addr 172.21.15.153:0/3282145662 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cfa59c640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 0x7f9cfc115250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 -- 172.21.15.153:0/3282145662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 msgr2=0x7f9cfc115250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 0x7f9cfc115250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 -- 172.21.15.153:0/3282145662 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec005040 con 0x7f9cfc0fd170 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.205+0000 7f9cf3fff640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4002a50 tx=0x7f9ce4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.206+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce4007c30 con 0x7f9cfc0fd170 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.206+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc10f4b0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.206+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ce400f040 con 0x7f9cfc0fd170 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.206+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ce400dad0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.206+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc10f9d0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.207+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc0fba60 con 0x7f9cfc0fd170 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.207+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9ce400dd40 con 0x7f9cfc0fd170 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.207+0000 7f9cf2ffd640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 0x7f9ce80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.208+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9ce4081c90 con 0x7f9cfc0fd170 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.208+0000 7f9cfa59c640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 0x7f9ce80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.208+0000 7f9cfa59c640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 0x7f9ce80678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009f80 tx=0x7f9cec035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:18.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.211+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9ce404cdf0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:18 smithi153 ceph-mon[29300]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:18.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:18 smithi173 ceph-mon[35650]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:18.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.380+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f9cfc10b6e0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.380+0000 7f9cf2ffd640 1 -- 172.21.15.153:0/3282145662 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9ce40501f0 con 0x7f9cfc0fd170 2024-06-06T03:15:18.380 INFO:teuthology.orchestra.run.smithi153.stdout:38654705700 2024-06-06T03:15:18.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 msgr2=0x7f9ce80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 0x7f9ce80678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009f80 tx=0x7f9cec035040 comp rx=0 tx=0).stop 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 msgr2=0x7f9cfc115790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4002a50 tx=0x7f9ce4004600 comp rx=0 tx=0).stop 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 shutdown_connections 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9cfc0fd170 0x7f9cfc115790 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ce8065430 0x7f9ce80678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 --2- 172.21.15.153:0/3282145662 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9cfc0faa50 0x7f9cfc115250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 >> 172.21.15.153:0/3282145662 conn(0x7f9cfc070ab0 msgr2=0x7f9cfc0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 shutdown_connections 2024-06-06T03:15:18.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.382+0000 7f9d00a1e640 1 -- 172.21.15.153:0/3282145662 wait complete. 2024-06-06T03:15:18.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 -- 172.21.15.153:0/3670859292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 msgr2=0x7f96140fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 --2- 172.21.15.153:0/3670859292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f96140fc2d0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0048d0 tx=0x7f95fc009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 -- 172.21.15.153:0/3670859292 shutdown_connections 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 --2- 172.21.15.153:0/3670859292 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f96140fc2d0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 --2- 172.21.15.153:0/3670859292 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 0x7f9614102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 -- 172.21.15.153:0/3670859292 >> 172.21.15.153:0/3670859292 conn(0x7f96140f7e70 msgr2=0x7f96140fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 -- 172.21.15.153:0/3670859292 shutdown_connections 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 -- 172.21.15.153:0/3670859292 wait complete. 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.863+0000 7f9619bf6640 1 Processor -- start 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 -- start start 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 0x7f961406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96140701f0 con 0x7f9614102c40 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9614070360 con 0x7f96141022a0 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:36194/0 (socket says 172.21.15.153:36194) 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 -- 172.21.15.153:0/2141100666 learned_addr learned my addr 172.21.15.153:0/2141100666 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f96137fe640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 0x7f961406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 -- 172.21.15.153:0/2141100666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 msgr2=0x7f961406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 0x7f961406f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 -- 172.21.15.153:0/2141100666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc005040 con 0x7f9614102c40 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f960bfff640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0043d0 tx=0x7f95fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95fc038070 con 0x7f9614102c40 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f95fc032970 con 0x7f9614102c40 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f95fc03da60 con 0x7f9614102c40 2024-06-06T03:15:18.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.864+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9614070590 con 0x7f9614102c40 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.865+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96141196b0 con 0x7f9614102c40 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.865+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96140fda10 con 0x7f9614102c40 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.866+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f95fc030020 con 0x7f9614102c40 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.866+0000 7f9610ff9640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.866+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f95fc0a2520 con 0x7f9614102c40 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.866+0000 7f96137fe640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.867+0000 7f96137fe640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 0x7f96000678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9604004ed0 tx=0x7f960400b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:18.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:18.870+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f95fc0709d0 con 0x7f9614102c40 2024-06-06T03:15:19.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.040+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f96140fba90 con 0x7f9614102c40 2024-06-06T03:15:19.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.040+0000 7f9610ff9640 1 -- 172.21.15.153:0/2141100666 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f95fc073ff0 con 0x7f9614102c40 2024-06-06T03:15:19.040 INFO:teuthology.orchestra.run.smithi153.stdout:146028888081 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 msgr2=0x7f96000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 0x7f96000678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9604004ed0 tx=0x7f960400b040 comp rx=0 tx=0).stop 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 msgr2=0x7f961406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f95fc0043d0 tx=0x7f95fc009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 shutdown_connections 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9614102c40 0x7f961406fcb0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:19.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9600065430 0x7f96000678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:19.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 --2- 172.21.15.153:0/2141100666 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f96141022a0 0x7f961406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:19.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.042+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 >> 172.21.15.153:0/2141100666 conn(0x7f96140f7e70 msgr2=0x7f96140f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:19.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.043+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 shutdown_connections 2024-06-06T03:15:19.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:19.043+0000 7f9619bf6640 1 -- 172.21.15.153:0/2141100666 wait complete. 2024-06-06T03:15:19.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:19 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3282145662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T03:15:19.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:19 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2141100666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T03:15:19.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:19 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3282145662' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-06T03:15:19.358 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:19 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2141100666' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-06T03:15:19.394 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-06-06T03:15:19.394 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:20.035 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888080 got 146028888081 for osd.4 2024-06-06T03:15:20.035 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:20.339 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:20 smithi173 ceph-mon[35650]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:20.485 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:20 smithi153 ceph-mon[29300]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:20.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705697 got 38654705700 for osd.0 2024-06-06T03:15:20.514 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:20.565 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- 172.21.15.153:0/1791170941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 msgr2=0x7f86f41085d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/1791170941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f41085d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f86e80048d0 tx=0x7f86e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- 172.21.15.153:0/1791170941 shutdown_connections 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/1791170941 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f41085d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/1791170941 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f406fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- 172.21.15.153:0/1791170941 >> 172.21.15.153:0/1791170941 conn(0x7f86f40f7e50 msgr2=0x7f86f40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- 172.21.15.153:0/1791170941 shutdown_connections 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- 172.21.15.153:0/1791170941 wait complete. 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 Processor -- start 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- start start 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f4115240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f4115d80 con 0x7f86f4070120 2024-06-06T03:15:22.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.180+0000 7f86f8a7f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f410f1c0 con 0x7f86f406f810 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41606/0 (socket says 172.21.15.153:41606) 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 -- 172.21.15.153:0/3107524627 learned_addr learned my addr 172.21.15.153:0/3107524627 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f259c640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f4115240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 -- 172.21.15.153:0/3107524627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 msgr2=0x7f86f4115240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f4115240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 -- 172.21.15.153:0/3107524627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e8005040 con 0x7f86f4070120 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f259c640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f4115240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f1d9b640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f86e8000c00 tx=0x7f86e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:22.188 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86e8038070 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f410f440 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f86e8008ee0 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f86e803c770 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.181+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f410f8a0 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.182+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f4070f50 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.183+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f86e8030020 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.183+0000 7f86e2ffd640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 0x7f86dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.183+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f86e80a2200 con 0x7f86f4070120 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.183+0000 7f86f259c640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 0x7f86dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:22.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.183+0000 7f86f259c640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 0x7f86dc0678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f86e40012a0 tx=0x7f86e40079f0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:22.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.188+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f86e8070770 con 0x7f86f4070120 2024-06-06T03:15:22.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:22 smithi153 ceph-mon[29300]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:22.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:22 smithi173 ceph-mon[35650]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.354+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f86f410b600 con 0x7f86f4070120 2024-06-06T03:15:22.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.354+0000 7f86e2ffd640 1 -- 172.21.15.153:0/3107524627 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f86e8073d90 con 0x7f86f4070120 2024-06-06T03:15:22.353 INFO:teuthology.orchestra.run.smithi153.stdout:167503724556 2024-06-06T03:15:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 msgr2=0x7f86dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 0x7f86dc0678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f86e40012a0 tx=0x7f86e40079f0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 msgr2=0x7f86f4115780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f86e8000c00 tx=0x7f86e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 shutdown_connections 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f86f4070120 0x7f86f4115780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f86dc065430 0x7f86dc0678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 --2- 172.21.15.153:0/3107524627 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f86f406f810 0x7f86f4115240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 >> 172.21.15.153:0/3107524627 conn(0x7f86f40f7e50 msgr2=0x7f86f40f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 shutdown_connections 2024-06-06T03:15:22.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:22.356+0000 7f86f8a7f640 1 -- 172.21.15.153:0/3107524627 wait complete. 2024-06-06T03:15:23.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724554 got 167503724556 for osd.5 2024-06-06T03:15:23.149 DEBUG:teuthology.parallel:result is None 2024-06-06T03:15:23.149 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-06T03:15:23.149 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T03:15:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:23.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:23 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3107524627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T03:15:23.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:23 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3107524627' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-06T03:15:24.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:24 smithi153 ceph-mon[29300]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:24 smithi173 ceph-mon[35650]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 -- 172.21.15.153:0/2673510130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e94101b40 msgr2=0x7f4e940fb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/2673510130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e94101b40 0x7f4e940fb1d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4e800048d0 tx=0x7f4e80009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 -- 172.21.15.153:0/2673510130 shutdown_connections 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/2673510130 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e94101b40 0x7f4e940fb1d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/2673510130 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e941011a0 0x7f4e94101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.168+0000 7f4e9ade9640 1 -- 172.21.15.153:0/2673510130 >> 172.21.15.153:0/2673510130 conn(0x7f4e940f6d90 msgr2=0x7f4e940f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 -- 172.21.15.153:0/2673510130 shutdown_connections 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 -- 172.21.15.153:0/2673510130 wait complete. 2024-06-06T03:15:25.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 Processor -- start 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 -- start start 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e94101b40 0x7f4e9410e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9410eb40 con 0x7f4e941011a0 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.169+0000 7f4e9ade9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9410ecb0 con 0x7f4e94101b40 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41626/0 (socket says 172.21.15.153:41626) 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 -- 172.21.15.153:0/4087776360 learned_addr learned my addr 172.21.15.153:0/4087776360 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e93fff640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e94101b40 0x7f4e9410e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 -- 172.21.15.153:0/4087776360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e94101b40 msgr2=0x7f4e9410e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e94101b40 0x7f4e9410e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 -- 172.21.15.153:0/4087776360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e80005040 con 0x7f4e941011a0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e98b85640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800c810 tx=0x7f4e880043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e8800a820 con 0x7f4e941011a0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4e8800ae50 con 0x7f4e941011a0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e94117020 con 0x7f4e941011a0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4e880115c0 con 0x7f4e941011a0 2024-06-06T03:15:25.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.170+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e94117530 con 0x7f4e941011a0 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.171+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e940fc910 con 0x7f4e941011a0 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.172+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4e8800a980 con 0x7f4e941011a0 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.172+0000 7f4e917fa640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 0x7f4e840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.172+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4e88082a30 con 0x7f4e941011a0 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.172+0000 7f4e93fff640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 0x7f4e840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:25.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.172+0000 7f4e93fff640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 0x7f4e840678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009f80 tx=0x7f4e80035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:25.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.176+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4e88050ee0 con 0x7f4e941011a0 2024-06-06T03:15:25.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.340+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e94065d20 con 0x7f4e84065430 2024-06-06T03:15:25.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.341+0000 7f4e917fa640 1 -- 172.21.15.153:0/4087776360 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27815 (secure 0 0 0) 0x7f4e94065d20 con 0x7f4e84065430 2024-06-06T03:15:25.341 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:25.341 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-06-06T03:15:25.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 msgr2=0x7f4e840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 0x7f4e840678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e80009f80 tx=0x7f4e80035040 comp rx=0 tx=0).stop 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 msgr2=0x7f4e9410dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4e8800c810 tx=0x7f4e880043c0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 shutdown_connections 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4e84065430 0x7f4e840678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f4e94101b40 0x7f4e9410e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 --2- 172.21.15.153:0/4087776360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4e941011a0 0x7f4e9410dfb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 >> 172.21.15.153:0/4087776360 conn(0x7f4e940f6d90 msgr2=0x7f4e940f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 shutdown_connections 2024-06-06T03:15:25.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:25.343+0000 7f4e9ade9640 1 -- 172.21.15.153:0/4087776360 wait complete. 2024-06-06T03:15:26.047 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-06-06T03:15:24.766487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3969024,"data_stored":2603946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002201"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T03:14:30.986521+0000","last_change":"2024-06-06T03:14:05.875499+0000","last_active":"2024-06-06T03:14:30.986521+0000","last_peered":"2024-06-06T03:14:30.986521+0000","last_clean":"2024-06-06T03:14:30.986521+0000","last_became_active":"2024-06-06T03:14:05.875169+0000","last_became_peered":"2024-06-06T03:14:05.875169+0000","last_unstale":"2024-06-06T03:14:30.986521+0000","last_undegraded":"2024-06-06T03:14:30.986521+0000","last_fullsized":"2024-06-06T03:14:30.986521+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T03:13:21.588468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T03:13:21.588468+0000","last_clean_scrub_stamp":"2024-06-06T03:13:21.588468+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T10:40:54.019154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.505}]},{"osd":2,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.443},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jun 6 03:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:14:20 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.318,"15min":0.318},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.30099999999999999}]},{"osd":1,"last update":"Thu Jun 6 03:14:20 2024","interfaces":[{"interface":"back","average":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.373}]},{"osd":3,"last update":"Thu Jun 6 03:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.625},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.63300000000000001}]},{"osd":4,"last update":"Thu Jun 6 03:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60999999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.30499999999999999,"15min":0.30099999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jun 6 03:14:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jun 6 03:14:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jun 6 03:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 03:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.252,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.31,"15min":0.30099999999999999},"min":{"1min":0.27100000000000002,"5min":0.191,"15min":0.191},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.246}]},{"osd":2,"last update":"Thu Jun 6 03:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jun 6 03:14:43 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jun 6 03:15:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T03:15:26.048 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-06-06T03:15:26.301 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:26.423 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:26 smithi153 ceph-mon[29300]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:26.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:26 smithi153 ceph-mon[29300]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T03:15:26.424 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:26.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:26 smithi173 ceph-mon[35650]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:26.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:26 smithi173 ceph-mon[35650]: from='client.14426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T03:15:26.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- 172.21.15.153:0/2015357097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640064d40 msgr2=0x7ff640065110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 --2- 172.21.15.153:0/2015357097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640064d40 0x7ff640065110 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff63c0048d0 tx=0x7ff63c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- 172.21.15.153:0/2015357097 shutdown_connections 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 --2- 172.21.15.153:0/2015357097 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640065650 0x7ff640108860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 --2- 172.21.15.153:0/2015357097 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640064d40 0x7ff640065110 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- 172.21.15.153:0/2015357097 >> 172.21.15.153:0/2015357097 conn(0x7ff640072110 msgr2=0x7ff640072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- 172.21.15.153:0/2015357097 shutdown_connections 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- 172.21.15.153:0/2015357097 wait complete. 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 Processor -- start 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- start start 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640064d40 0x7ff640115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff640115dc0 con 0x7ff640065650 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.989+0000 7ff6478c8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff64010f250 con 0x7ff640064d40 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:41652/0 (socket says 172.21.15.153:41652) 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 -- 172.21.15.153:0/606990561 learned_addr learned my addr 172.21.15.153:0/606990561 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff645664640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640064d40 0x7ff640115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 -- 172.21.15.153:0/606990561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640064d40 msgr2=0x7ff640115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:28.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640064d40 0x7ff640115280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 -- 172.21.15.153:0/606990561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63c005040 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff644e63640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff630002840 tx=0x7ff630004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff630007c30 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff63000f040 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff64010f530 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff63000da30 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.990+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff64010fa50 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.991+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6400fb830 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.992+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff6300096a0 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.992+0000 7ff635ffb640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.992+0000 7ff645664640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.992+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff630082250 con 0x7ff640065650 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.993+0000 7ff645664640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 0x7ff62c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009f80 tx=0x7ff63c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:28.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:27.996+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff630050840 con 0x7ff640065650 2024-06-06T03:15:28.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.158+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 --> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6401108f0 con 0x7ff62c065430 2024-06-06T03:15:28.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.160+0000 7ff635ffb640 1 -- 172.21.15.153:0/606990561 <== mgr.14212 v2:172.21.15.153:6800/2373762556 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27856 (secure 0 0 0) 0x7ff6401108f0 con 0x7ff62c065430 2024-06-06T03:15:28.159 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:28.160 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.161+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 msgr2=0x7ff62c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.161+0000 7ff6478c8640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 0x7ff62c0678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff63c009f80 tx=0x7ff63c035040 comp rx=0 tx=0).stop 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.161+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 msgr2=0x7ff6401157c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.161+0000 7ff6478c8640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff630002840 tx=0x7ff630004570 comp rx=0 tx=0).stop 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 shutdown_connections 2024-06-06T03:15:28.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff640065650 0x7ff6401157c0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7ff62c065430 0x7ff62c0678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 --2- 172.21.15.153:0/606990561 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7ff640064d40 0x7ff640115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:28.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 >> 172.21.15.153:0/606990561 conn(0x7ff640072110 msgr2=0x7ff640103820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:28.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 shutdown_connections 2024-06-06T03:15:28.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:28.162+0000 7ff6478c8640 1 -- 172.21.15.153:0/606990561 wait complete. 2024-06-06T03:15:28.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:28 smithi153 ceph-mon[29300]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:28.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:28 smithi173 ceph-mon[35650]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:28.801 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-06-06T03:15:26.766834+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788028,"kb_used_data":3876,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682884,"statfs":{"total":575970213888,"available":574139273216,"internally_reserved":0,"allocated":3969024,"data_stored":2603946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002278"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":189,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-06-06T03:14:30.986521+0000","last_change":"2024-06-06T03:14:05.875499+0000","last_active":"2024-06-06T03:14:30.986521+0000","last_peered":"2024-06-06T03:14:30.986521+0000","last_clean":"2024-06-06T03:14:30.986521+0000","last_became_active":"2024-06-06T03:14:05.875169+0000","last_became_peered":"2024-06-06T03:14:05.875169+0000","last_unstale":"2024-06-06T03:14:30.986521+0000","last_undegraded":"2024-06-06T03:14:30.986521+0000","last_fullsized":"2024-06-06T03:14:30.986521+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-06T03:13:21.588468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-06T03:13:21.588468+0000","last_clean_scrub_stamp":"2024-06-06T03:13:21.588468+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-07T10:40:54.019154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":39,"seq":167503724557,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35099999999999998}]}]},{"osd":4,"up_from":34,"seq":146028888083,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297716,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447436,"statfs":{"total":95995035648,"available":95690174464,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.59299999999999997}]},{"osd":1,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.438,"15min":0.438},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.45900000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.55000000000000004}]},{"osd":2,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.56299999999999994}]},{"osd":3,"last update":"Thu Jun 6 03:15:06 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.53300000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]},{"osd":3,"up_from":26,"seq":111669149719,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298300,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446852,"statfs":{"total":95995035648,"available":95689576448,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.49099999999999999}]},{"osd":1,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.505}]},{"osd":2,"last update":"Thu Jun 6 03:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.443},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Thu Jun 6 03:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.377,"15min":0.377},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.33800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297712,"kb_used_data":356,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447440,"statfs":{"total":95995035648,"available":95690178560,"internally_reserved":0,"allocated":364544,"data_stored":138807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.27400000000000002,"5min":0.309,"15min":0.315},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.39400000000000002,"5min":0.442,"15min":0.442},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.29999999999999999,"5min":0.33100000000000002,"15min":0.33600000000000002},"min":{"1min":0.23100000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.42699999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.433}]},{"osd":1,"last update":"Thu Jun 6 03:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.29499999999999998,"5min":0.32700000000000001,"15min":0.33300000000000002},"min":{"1min":0.23699999999999999,"5min":0.224,"15min":0.224},"max":{"1min":0.39800000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45700000000000002,"5min":0.502,"15min":0.502},"last":0.443}]},{"osd":3,"last update":"Thu Jun 6 03:14:47 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.59699999999999998}]},{"osd":4,"last update":"Thu Jun 6 03:15:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.64300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298292,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446860,"statfs":{"total":95995035648,"available":95689584640,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jun 6 03:14:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.30499999999999999,"15min":0.30099999999999999},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.502,"5min":0.502,"15min":0.502},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.318,"15min":0.312},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"last":0.52700000000000002}]},{"osd":2,"last update":"Thu Jun 6 03:14:20 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Jun 6 03:14:43 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.68700000000000006}]},{"osd":4,"last update":"Thu Jun 6 03:15:09 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298296,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446856,"statfs":{"total":95995035648,"available":95689580544,"internally_reserved":0,"allocated":958464,"data_stored":729175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jun 6 03:14:52 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.31,"15min":0.30199999999999999},"min":{"1min":0.252,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.31,"15min":0.30099999999999999},"min":{"1min":0.27100000000000002,"5min":0.191,"15min":0.191},"max":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"last":0.246}]},{"osd":2,"last update":"Thu Jun 6 03:14:23 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.26400000000000001}]},{"osd":3,"last update":"Thu Jun 6 03:14:43 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.44400000000000001}]},{"osd":4,"last update":"Thu Jun 6 03:15:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.49399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.433}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-06T03:15:28.802 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-06T03:15:28.802 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-06T03:15:28.802 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-06T03:15:28.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:29.057 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:29.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:29 smithi153 ceph-mon[29300]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T03:15:29.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:29 smithi173 ceph-mon[35650]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-06T03:15:30.372 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:30 smithi153 ceph-mon[29300]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:30.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:30 smithi173 ceph-mon[35650]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:30.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 -- 172.21.15.153:0/4283181163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc101b60 msgr2=0x7f18fc0fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:30.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 --2- 172.21.15.153:0/4283181163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc101b60 0x7f18fc0fb1f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f18e80048d0 tx=0x7f18e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:30.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 -- 172.21.15.153:0/4283181163 shutdown_connections 2024-06-06T03:15:30.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 --2- 172.21.15.153:0/4283181163 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc101b60 0x7f18fc0fb1f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 --2- 172.21.15.153:0/4283181163 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc1011c0 0x7f18fc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 -- 172.21.15.153:0/4283181163 >> 172.21.15.153:0/4283181163 conn(0x7f18fc0f6d70 msgr2=0x7f18fc0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 -- 172.21.15.153:0/4283181163 shutdown_connections 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.920+0000 7f1902c45640 1 -- 172.21.15.153:0/4283181163 wait complete. 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 Processor -- start 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 -- start start 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc101b60 0x7f18fc112460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc112a60 con 0x7f18fc1011c0 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f1902c45640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18fc10bea0 con 0x7f18fc101b60 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.921+0000 7f19009e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60736/0 (socket says 172.21.15.153:60736) 2024-06-06T03:15:30.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 -- 172.21.15.153:0/1429989695 learned_addr learned my addr 172.21.15.153:0/1429989695 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f18fbfff640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc101b60 0x7f18fc112460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 -- 172.21.15.153:0/1429989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc101b60 msgr2=0x7f18fc112460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc101b60 0x7f18fc112460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 -- 172.21.15.153:0/1429989695 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e8005040 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f19009e1640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f18f0002a50 tx=0x7f18f0004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f0007c30 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18f000f040 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18fc10c180 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f18f000dad0 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.922+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18fc10c6f0 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.923+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18fc0fc930 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.924+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f18f000dc70 con 0x7f18fc1011c0 2024-06-06T03:15:30.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.924+0000 7f18f97fa640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 0x7f18ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:30.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.924+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f18f0081bb0 con 0x7f18fc1011c0 2024-06-06T03:15:30.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.925+0000 7f18fbfff640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 0x7f18ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:30.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.925+0000 7f18fbfff640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 0x7f18ec0678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f80 tx=0x7f18e8035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:30.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:30.929+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f18f004cdf0 con 0x7f18fc1011c0 2024-06-06T03:15:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.128+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f18fc04b480 con 0x7f18fc1011c0 2024-06-06T03:15:31.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.128+0000 7f18f97fa640 1 -- 172.21.15.153:0/1429989695 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f18f0088080 con 0x7f18fc1011c0 2024-06-06T03:15:31.128 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:31.128 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 msgr2=0x7f18ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 0x7f18ec0678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f80 tx=0x7f18e8035040 comp rx=0 tx=0).stop 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 msgr2=0x7f18fc111f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f18f0002a50 tx=0x7f18f0004600 comp rx=0 tx=0).stop 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 shutdown_connections 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f18ec065430 0x7f18ec0678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f18fc101b60 0x7f18fc112460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 --2- 172.21.15.153:0/1429989695 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18fc1011c0 0x7f18fc111f20 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 >> 172.21.15.153:0/1429989695 conn(0x7f18fc0f6d70 msgr2=0x7f18fc0f7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 shutdown_connections 2024-06-06T03:15:31.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:31.130+0000 7f1902c45640 1 -- 172.21.15.153:0/1429989695 wait complete. 2024-06-06T03:15:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:32 smithi153 ceph-mon[29300]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:32 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1429989695' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:32 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:15:32.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:32 smithi173 ceph-mon[35650]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:32.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:32 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1429989695' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:32.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:32 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:15:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:15:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:15:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:15:33.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:15:33.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:15:33.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:15:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:34 smithi153 ceph-mon[29300]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:34.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:34 smithi173 ceph-mon[35650]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:34.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:35.192 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:36.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:36 smithi153 ceph-mon[29300]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:36.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:36 smithi173 ceph-mon[35650]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 -- 172.21.15.153:0/925228758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 msgr2=0x7fbf1006f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 --2- 172.21.15.153:0/925228758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf1006f810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0048d0 tx=0x7fbf0c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 -- 172.21.15.153:0/925228758 shutdown_connections 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 --2- 172.21.15.153:0/925228758 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf100701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 --2- 172.21.15.153:0/925228758 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf1006f810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 -- 172.21.15.153:0/925228758 >> 172.21.15.153:0/925228758 conn(0x7fbf100f7e50 msgr2=0x7fbf100fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 -- 172.21.15.153:0/925228758 shutdown_connections 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.948+0000 7fbf1729f640 1 -- 172.21.15.153:0/925228758 wait complete. 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 Processor -- start 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 -- start start 2024-06-06T03:15:36.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf10119990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1011a520 con 0x7fbf10071440 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1729f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10113980 con 0x7fbf1006fd50 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1483a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1483a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60764/0 (socket says 172.21.15.153:60764) 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.949+0000 7fbf1483a640 1 -- 172.21.15.153:0/2178995437 learned_addr learned my addr 172.21.15.153:0/2178995437 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1503b640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf10119990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1483a640 1 -- 172.21.15.153:0/2178995437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 msgr2=0x7fbf10119990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1483a640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf10119990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1483a640 1 -- 172.21.15.153:0/2178995437 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c005040 con 0x7fbf10071440 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1503b640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf10119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1483a640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf00002a50 tx=0x7fbf00004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf00007c30 con 0x7fbf10071440 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf10113c60 con 0x7fbf10071440 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbf0000f040 con 0x7fbf10071440 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbf0000db50 con 0x7fbf10071440 2024-06-06T03:15:36.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.950+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf10114180 con 0x7fbf10071440 2024-06-06T03:15:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.951+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf10070f50 con 0x7fbf10071440 2024-06-06T03:15:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.951+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbf0001c400 con 0x7fbf10071440 2024-06-06T03:15:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.952+0000 7fbf05ffb640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 0x7fbefc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.952+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbf00081e00 con 0x7fbf10071440 2024-06-06T03:15:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.952+0000 7fbf1503b640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 0x7fbefc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.952+0000 7fbf1503b640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 0x7fbefc0678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0043d0 tx=0x7fbf0c009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:36.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:36.956+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbf000502b0 con 0x7fbf10071440 2024-06-06T03:15:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.153+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbf100fbab0 con 0x7fbf10071440 2024-06-06T03:15:37.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.153+0000 7fbf05ffb640 1 -- 172.21.15.153:0/2178995437 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbf000538d0 con 0x7fbf10071440 2024-06-06T03:15:37.154 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:37.154 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 msgr2=0x7fbefc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 0x7fbefc0678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0043d0 tx=0x7fbf0c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 msgr2=0x7fbf10119ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbf00002a50 tx=0x7fbf00004600 comp rx=0 tx=0).stop 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 shutdown_connections 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbf10071440 0x7fbf10119ed0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbefc065430 0x7fbefc0678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 --2- 172.21.15.153:0/2178995437 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbf1006fd50 0x7fbf10119990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 >> 172.21.15.153:0/2178995437 conn(0x7fbf100f7e50 msgr2=0x7fbf100f90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 shutdown_connections 2024-06-06T03:15:37.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:37.155+0000 7fbf1729f640 1 -- 172.21.15.153:0/2178995437 wait complete. 2024-06-06T03:15:37.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:37 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2178995437' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:37.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:37 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2178995437' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:38.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:38 smithi153 ceph-mon[29300]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:38.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:38 smithi173 ceph-mon[35650]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:40.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:40 smithi153 ceph-mon[29300]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:40.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:40 smithi173 ceph-mon[35650]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:40.954 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:41.221 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:41.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:41.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:42.432 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:42 smithi153 ceph-mon[29300]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:42.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:42 smithi173 ceph-mon[35650]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 -- 172.21.15.153:0/3504348460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 msgr2=0x7f3c440fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/3504348460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c440fb1e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3c300048d0 tx=0x7f3c30009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 -- 172.21.15.153:0/3504348460 shutdown_connections 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/3504348460 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c440fb1e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/3504348460 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 0x7f3c44101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 -- 172.21.15.153:0/3504348460 >> 172.21.15.153:0/3504348460 conn(0x7f3c440f6d60 msgr2=0x7f3c440f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 -- 172.21.15.153:0/3504348460 shutdown_connections 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.935+0000 7f3c4ac92640 1 -- 172.21.15.153:0/3504348460 wait complete. 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.936+0000 7f3c4ac92640 1 Processor -- start 2024-06-06T03:15:42.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.936+0000 7f3c4ac92640 1 -- start start 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.936+0000 7f3c4ac92640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 0x7f3c44111f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c4ac92640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c4ac92640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c44112aa0 con 0x7f3c44101b50 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c4ac92640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4410bee0 con 0x7f3c441011b0 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53668/0 (socket says 172.21.15.153:53668) 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 -- 172.21.15.153:0/2008224799 learned_addr learned my addr 172.21.15.153:0/2008224799 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c48a2e640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 0x7f3c44111f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 -- 172.21.15.153:0/2008224799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 msgr2=0x7f3c44111f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 0x7f3c44111f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 -- 172.21.15.153:0/2008224799 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c30005040 con 0x7f3c44101b50 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c43fff640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3c300043d0 tx=0x7f3c30009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:42.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c30038070 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c30032970 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c4410c160 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c3003da60 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.937+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c4410c6a0 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.938+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c440fc920 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.939+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3c30030020 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.939+0000 7f3c417fa640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 0x7f3c340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.939+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3c300a25c0 con 0x7f3c44101b50 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.939+0000 7f3c48a2e640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 0x7f3c340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.940+0000 7f3c48a2e640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 0x7f3c340678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c38004ed0 tx=0x7f3c3800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:42.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:42.943+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3c30070a70 con 0x7f3c44101b50 2024-06-06T03:15:43.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.142+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c44108310 con 0x7f3c44101b50 2024-06-06T03:15:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.142+0000 7f3c417fa640 1 -- 172.21.15.153:0/2008224799 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3c30074090 con 0x7f3c44101b50 2024-06-06T03:15:43.143 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:43.143 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 msgr2=0x7f3c340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 0x7f3c340678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3c38004ed0 tx=0x7f3c3800b040 comp rx=0 tx=0).stop 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 msgr2=0x7f3c441124a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f3c300043d0 tx=0x7f3c30009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 shutdown_connections 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3c44101b50 0x7f3c441124a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3c34065430 0x7f3c340678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 --2- 172.21.15.153:0/2008224799 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3c441011b0 0x7f3c44111f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 >> 172.21.15.153:0/2008224799 conn(0x7f3c440f6d60 msgr2=0x7f3c440f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 shutdown_connections 2024-06-06T03:15:43.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:43.144+0000 7f3c4ac92640 1 -- 172.21.15.153:0/2008224799 wait complete. 2024-06-06T03:15:43.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:43 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2008224799' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:43.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:43 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2008224799' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:44.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:44 smithi153 ceph-mon[29300]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:44.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:44 smithi173 ceph-mon[35650]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:46 smithi153 ceph-mon[29300]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:46.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:46 smithi173 ceph-mon[35650]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:46.757 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:47.023 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:48.461 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:48 smithi153 ceph-mon[29300]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:48.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:48 smithi173 ceph-mon[35650]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:48.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- 172.21.15.153:0/2476059160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 msgr2=0x7fa9780faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 --2- 172.21.15.153:0/2476059160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9780faec0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa96c0048d0 tx=0x7fa96c009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- 172.21.15.153:0/2476059160 shutdown_connections 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 --2- 172.21.15.153:0/2476059160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9780faec0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 --2- 172.21.15.153:0/2476059160 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 0x7fa9781027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- 172.21.15.153:0/2476059160 >> 172.21.15.153:0/2476059160 conn(0x7fa978070ab0 msgr2=0x7fa978070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- 172.21.15.153:0/2476059160 shutdown_connections 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- 172.21.15.153:0/2476059160 wait complete. 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 Processor -- start 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.708+0000 7fa97caab640 1 -- start start 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 0x7fa978113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:48.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa978113c20 con 0x7fa9780faa50 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa97810d060 con 0x7fa9781023e0 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53680/0 (socket says 172.21.15.153:53680) 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 -- 172.21.15.153:0/3934472215 learned_addr learned my addr 172.21.15.153:0/3934472215 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa975d9b640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 0x7fa978113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 -- 172.21.15.153:0/3934472215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 msgr2=0x7fa978113620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 0x7fa978113620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 -- 172.21.15.153:0/3934472215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c005040 con 0x7fa9780faa50 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97659c640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa968002a50 tx=0x7fa968004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa968007c30 con 0x7fa9780faa50 2024-06-06T03:15:48.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97810d340 con 0x7fa9780faa50 2024-06-06T03:15:48.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97810d830 con 0x7fa9780faa50 2024-06-06T03:15:48.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa96800f040 con 0x7fa9780faa50 2024-06-06T03:15:48.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.709+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96800dad0 con 0x7fa9780faa50 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.710+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa978065550 con 0x7fa9780faa50 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.711+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa96800dd10 con 0x7fa9780faa50 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.711+0000 7fa966ffd640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 0x7fa9600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.711+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa968081c90 con 0x7fa9780faa50 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.711+0000 7fa975d9b640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 0x7fa9600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:48.714 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.712+0000 7fa975d9b640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 0x7fa9600678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009f80 tx=0x7fa96c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:48.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.715+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa96804cdf0 con 0x7fa9780faa50 2024-06-06T03:15:48.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.915+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa97804b480 con 0x7fa9780faa50 2024-06-06T03:15:48.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.915+0000 7fa966ffd640 1 -- 172.21.15.153:0/3934472215 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa968050330 con 0x7fa9780faa50 2024-06-06T03:15:48.916 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:48.916 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:15:48.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 msgr2=0x7fa9600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 0x7fa9600678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa96c009f80 tx=0x7fa96c035040 comp rx=0 tx=0).stop 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 msgr2=0x7fa9781130e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa968002a50 tx=0x7fa968004600 comp rx=0 tx=0).stop 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 shutdown_connections 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fa960065430 0x7fa9600678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fa9781023e0 0x7fa978113620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 --2- 172.21.15.153:0/3934472215 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa9780faa50 0x7fa9781130e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 >> 172.21.15.153:0/3934472215 conn(0x7fa978070ab0 msgr2=0x7fa9780f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 shutdown_connections 2024-06-06T03:15:48.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:48.917+0000 7fa97caab640 1 -- 172.21.15.153:0/3934472215 wait complete. 2024-06-06T03:15:49.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:49 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3934472215' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:49.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:49 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3934472215' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:50.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:50 smithi153 ceph-mon[29300]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:50.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:50 smithi173 ceph-mon[35650]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:52.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:52 smithi153 ceph-mon[29300]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:52.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:52.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:52 smithi173 ceph-mon[35650]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:52.849 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 -- 172.21.15.153:0/2433438754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 msgr2=0x7f12941015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 --2- 172.21.15.153:0/2433438754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f12941015a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f12840048d0 tx=0x7f1284009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 -- 172.21.15.153:0/2433438754 shutdown_connections 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 --2- 172.21.15.153:0/2433438754 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 0x7f12940fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 --2- 172.21.15.153:0/2433438754 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f12941015a0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 -- 172.21.15.153:0/2433438754 >> 172.21.15.153:0/2433438754 conn(0x7f12940f6dc0 msgr2=0x7f12940f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 -- 172.21.15.153:0/2433438754 shutdown_connections 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 -- 172.21.15.153:0/2433438754 wait complete. 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.487+0000 7f1299e88640 1 Processor -- start 2024-06-06T03:15:54.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1299e88640 1 -- start start 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1299e88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1299e88640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 0x7f129410e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1299e88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129410eae0 con 0x7f12941011d0 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1299e88640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129410ec50 con 0x7f1294101b70 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47968/0 (socket says 172.21.15.153:47968) 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 -- 172.21.15.153:0/834904477 learned_addr learned my addr 172.21.15.153:0/834904477 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f1292ffd640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 0x7f129410e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 -- 172.21.15.153:0/834904477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 msgr2=0x7f129410e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 0x7f129410e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 -- 172.21.15.153:0/834904477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284005040 con 0x7f12941011d0 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f12937fe640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1284003080 tx=0x7f1284009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:54.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.488+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1284038070 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.489+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1294117010 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.489+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1294117470 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.489+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1284008ee0 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.489+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f128403d870 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.490+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12940fc940 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.490+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1284030020 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.490+0000 7f127bfff640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 0x7f127c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.490+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f12840a21a0 con 0x7f12941011d0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.490+0000 7f1292ffd640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 0x7f127c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:15:54.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.491+0000 7f1292ffd640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 0x7f127c0678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12940661f0 tx=0x7f1288009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:15:54.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.495+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1284070770 con 0x7f12941011d0 2024-06-06T03:15:54.495 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:54 smithi153 ceph-mon[29300]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:54.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:54 smithi173 ceph-mon[35650]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.691+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f12940fa9c0 con 0x7f12941011d0 2024-06-06T03:15:54.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.692+0000 7f127bfff640 1 -- 172.21.15.153:0/834904477 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1284073d90 con 0x7f12941011d0 2024-06-06T03:15:54.691 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:15:54.691 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:15:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 msgr2=0x7f127c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:54.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 0x7f127c0678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f12940661f0 tx=0x7f1288009040 comp rx=0 tx=0).stop 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 msgr2=0x7f129410dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1284003080 tx=0x7f1284009da0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 shutdown_connections 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f127c065430 0x7f127c0678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1294101b70 0x7f129410e4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 --2- 172.21.15.153:0/834904477 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12941011d0 0x7f129410dfa0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 >> 172.21.15.153:0/834904477 conn(0x7f12940f6dc0 msgr2=0x7f12940f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 shutdown_connections 2024-06-06T03:15:54.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:15:54.694+0000 7f1299e88640 1 -- 172.21.15.153:0/834904477 wait complete. 2024-06-06T03:15:55.331 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:55 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/834904477' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:55.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:55 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/834904477' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:15:56.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:56 smithi173 ceph-mon[35650]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:56.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:56.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:56 smithi153 ceph-mon[29300]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:56.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:15:58.356 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:15:58.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:15:58 smithi173 ceph-mon[35650]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:15:58.621 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:15:58.743 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:15:58 smithi153 ceph-mon[29300]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:00.314 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:00 smithi153 ceph-mon[29300]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:00.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 -- 172.21.15.153:0/4233467065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 msgr2=0x7f81f40fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:00.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 --2- 172.21.15.153:0/4233467065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f40fb940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f81e80048d0 tx=0x7f81e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 -- 172.21.15.153:0/4233467065 shutdown_connections 2024-06-06T03:16:00.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 --2- 172.21.15.153:0/4233467065 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 0x7f81f40fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 --2- 172.21.15.153:0/4233467065 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f40fb940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 -- 172.21.15.153:0/4233467065 >> 172.21.15.153:0/4233467065 conn(0x7f81f4072110 msgr2=0x7f81f4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 -- 172.21.15.153:0/4233467065 shutdown_connections 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.392+0000 7f81fac15640 1 -- 172.21.15.153:0/4233467065 wait complete. 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 Processor -- start 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 -- start start 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 0x7f81f410e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f410eb00 con 0x7f81f40fb570 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81fac15640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f410ec70 con 0x7f81f40fbe80 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f89b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f89b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54406/0 (socket says 172.21.15.153:54406) 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f89b1640 1 -- 172.21.15.153:0/729560608 learned_addr learned my addr 172.21.15.153:0/729560608 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:00.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f3fff640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 0x7f81f410e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f89b1640 1 -- 172.21.15.153:0/729560608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 msgr2=0x7f81f410e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.393+0000 7f81f89b1640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 0x7f81f410e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81f89b1640 1 -- 172.21.15.153:0/729560608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e8005040 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81f89b1640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f81e80043d0 tx=0x7f81e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e8038070 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f4117030 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81e8032970 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e803da60 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.394+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f4117490 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.395+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f40fc920 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.395+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f81e8030020 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.396+0000 7f81f17fa640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 0x7f81e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.396+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f81e80a25c0 con 0x7f81f40fb570 2024-06-06T03:16:00.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.396+0000 7f81f3fff640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 0x7f81e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:00.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.396+0000 7f81f3fff640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 0x7f81e40678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81f4065700 tx=0x7f81e000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:00.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.400+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f81e8070a70 con 0x7f81f40fb570 2024-06-06T03:16:00.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:00 smithi173 ceph-mon[35650]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.595+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f81f40654b0 con 0x7f81f40fb570 2024-06-06T03:16:00.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.596+0000 7f81f17fa640 1 -- 172.21.15.153:0/729560608 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f81e8074090 con 0x7f81f40fb570 2024-06-06T03:16:00.595 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:00.595 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:00.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 msgr2=0x7f81e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 0x7f81e40678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f81f4065700 tx=0x7f81e000b040 comp rx=0 tx=0).stop 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 msgr2=0x7f81f410dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f81e80043d0 tx=0x7f81e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 shutdown_connections 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81e4065430 0x7f81e40678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f40fbe80 0x7f81f410e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 --2- 172.21.15.153:0/729560608 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f40fb570 0x7f81f410dfc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 >> 172.21.15.153:0/729560608 conn(0x7f81f4072110 msgr2=0x7f81f40f86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 shutdown_connections 2024-06-06T03:16:00.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:00.598+0000 7f81fac15640 1 -- 172.21.15.153:0/729560608 wait complete. 2024-06-06T03:16:01.556 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/729560608' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:01.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:01 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/729560608' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:02.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:02 smithi173 ceph-mon[35650]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:02.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:02 smithi153 ceph-mon[29300]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:04.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:04.569 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:04.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:04 smithi173 ceph-mon[35650]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:04.691 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:04 smithi153 ceph-mon[29300]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:06.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 -- 172.21.15.153:0/576046786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fbe80 msgr2=0x7fd8e00fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:06.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/576046786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fbe80 0x7fd8e00fcd80 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd8c80048d0 tx=0x7fd8c8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 -- 172.21.15.153:0/576046786 shutdown_connections 2024-06-06T03:16:06.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/576046786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fbe80 0x7fd8e00fcd80 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/576046786 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fb570 0x7fd8e00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 -- 172.21.15.153:0/576046786 >> 172.21.15.153:0/576046786 conn(0x7fd8e0072110 msgr2=0x7fd8e0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 -- 172.21.15.153:0/576046786 shutdown_connections 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.289+0000 7fd8e66c9640 1 -- 172.21.15.153:0/576046786 wait complete. 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 Processor -- start 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 -- start start 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fbe80 0x7fd8e01123f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e01129f0 con 0x7fd8e00fb570 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8e66c9640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e010be30 con 0x7fd8e00fbe80 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54428/0 (socket says 172.21.15.153:54428) 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 -- 172.21.15.153:0/3247154550 learned_addr learned my addr 172.21.15.153:0/3247154550 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8d7fff640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fbe80 0x7fd8e01123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:06.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 -- 172.21.15.153:0/3247154550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fbe80 msgr2=0x7fd8e01123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fbe80 0x7fd8e01123f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 -- 172.21.15.153:0/3247154550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c8005040 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.290+0000 7fd8dffff640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000c830 tx=0x7fd8d0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.291+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8d000a850 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.291+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd8d000ae80 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.291+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e010c110 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.291+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd8d0012ea0 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.291+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e010c6b0 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.292+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e00fc920 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.292+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd8d0012400 con 0x7fd8e00fb570 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.292+0000 7fd8dd7fa640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 0x7fd8cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.293+0000 7fd8d7fff640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 0x7fd8cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:06.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.293+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd8d0082730 con 0x7fd8e00fb570 2024-06-06T03:16:06.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.293+0000 7fd8d7fff640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 0x7fd8cc0678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8000c00 tx=0x7fd8c8009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:06.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.297+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd8d0050c60 con 0x7fd8e00fb570 2024-06-06T03:16:06.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.496+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd8e0072c20 con 0x7fd8e00fb570 2024-06-06T03:16:06.496 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:06 smithi153 ceph-mon[29300]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:06.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.496+0000 7fd8dd7fa640 1 -- 172.21.15.153:0/3247154550 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd8d00126b0 con 0x7fd8e00fb570 2024-06-06T03:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:06.497 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:06.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.498+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 msgr2=0x7fd8cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.498+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 0x7fd8cc0678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8000c00 tx=0x7fd8c8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.498+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 msgr2=0x7fd8e0111eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.498+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd8d000c830 tx=0x7fd8d0004430 comp rx=0 tx=0).stop 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 shutdown_connections 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd8cc065430 0x7fd8cc0678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd8e00fbe80 0x7fd8e01123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 --2- 172.21.15.153:0/3247154550 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd8e00fb570 0x7fd8e0111eb0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 >> 172.21.15.153:0/3247154550 conn(0x7fd8e0072110 msgr2=0x7fd8e00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 shutdown_connections 2024-06-06T03:16:06.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:06.499+0000 7fd8e66c9640 1 -- 172.21.15.153:0/3247154550 wait complete. 2024-06-06T03:16:06.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:06 smithi173 ceph-mon[35650]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:07.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:07 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3247154550' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:07.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:07 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3247154550' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:08.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:08 smithi153 ceph-mon[29300]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:08.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:08 smithi173 ceph-mon[35650]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:10.146 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:10.414 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:10.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:10 smithi153 ceph-mon[29300]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:10.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:10 smithi173 ceph-mon[35650]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:11.561 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:11.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:12.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 -- 172.21.15.153:0/912581391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc1022f0 msgr2=0x7f5edc1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 --2- 172.21.15.153:0/912581391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc1022f0 0x7f5edc1026c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00048d0 tx=0x7f5ed0009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 -- 172.21.15.153:0/912581391 shutdown_connections 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 --2- 172.21.15.153:0/912581391 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc102c90 0x7f5edc0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 --2- 172.21.15.153:0/912581391 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc1022f0 0x7f5edc1026c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 -- 172.21.15.153:0/912581391 >> 172.21.15.153:0/912581391 conn(0x7f5edc070a40 msgr2=0x7f5edc070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 -- 172.21.15.153:0/912581391 shutdown_connections 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 -- 172.21.15.153:0/912581391 wait complete. 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.085+0000 7f5ee3263640 1 Processor -- start 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee3263640 1 -- start start 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee3263640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 0x7f5edc115270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:12.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee3263640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee3263640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc115db0 con 0x7f5edc102c90 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee3263640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc115f20 con 0x7f5edc1022f0 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45494/0 (socket says 172.21.15.153:45494) 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 -- 172.21.15.153:0/2475785385 learned_addr learned my addr 172.21.15.153:0/2475785385 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee0fff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 0x7f5edc115270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 -- 172.21.15.153:0/2475785385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 msgr2=0x7f5edc115270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 0x7f5edc115270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5edbfff640 1 -- 172.21.15.153:0/2475785385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ed0005040 con 0x7f5edc102c90 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.086+0000 7f5ee0fff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 0x7f5edc115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5edbfff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8002840 tx=0x7f5ec8004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec8007c30 con 0x7f5edc102c90 2024-06-06T03:16:12.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc10f460 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ec800f040 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ec8009570 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.087+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc10f980 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.088+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc065510 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.088+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5ec800dc80 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.089+0000 7f5ed97fa640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 0x7f5ecc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.089+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5ec8081da0 con 0x7f5edc102c90 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.089+0000 7f5ee0fff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 0x7f5ecc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:12.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.089+0000 7f5ee0fff640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 0x7f5ecc0678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000c00 tx=0x7f5ed0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:12.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.094+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5ec8050310 con 0x7f5edc102c90 2024-06-06T03:16:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.293+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5edc074960 con 0x7f5edc102c90 2024-06-06T03:16:12.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.294+0000 7f5ed97fa640 1 -- 172.21.15.153:0/2475785385 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5ec8053930 con 0x7f5edc102c90 2024-06-06T03:16:12.294 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:12.294 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 msgr2=0x7f5ecc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:12.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 0x7f5ecc0678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5ed0000c00 tx=0x7f5ed0035040 comp rx=0 tx=0).stop 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 msgr2=0x7f5edc1157b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5ec8002840 tx=0x7f5ec8004570 comp rx=0 tx=0).stop 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 shutdown_connections 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5edc102c90 0x7f5edc1157b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5ecc065430 0x7f5ecc0678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 --2- 172.21.15.153:0/2475785385 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5edc1022f0 0x7f5edc115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 >> 172.21.15.153:0/2475785385 conn(0x7f5edc070a40 msgr2=0x7f5edc1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 shutdown_connections 2024-06-06T03:16:12.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:12.296+0000 7f5ee3263640 1 -- 172.21.15.153:0/2475785385 wait complete. 2024-06-06T03:16:12.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:12 smithi173 ceph-mon[35650]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:12.592 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:12 smithi153 ceph-mon[29300]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:13.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:13 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2475785385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:13.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:13 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2475785385' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:14.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:14 smithi173 ceph-mon[35650]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:14.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:14 smithi153 ceph-mon[29300]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:15.928 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:16.192 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:16.579 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:16 smithi153 ceph-mon[29300]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:16.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:16 smithi173 ceph-mon[35650]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:17.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- 172.21.15.153:0/2351403193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 msgr2=0x7f50d01085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:17.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 --2- 172.21.15.153:0/2351403193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d01085f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f50c40048d0 tx=0x7f50c4009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- 172.21.15.153:0/2351403193 shutdown_connections 2024-06-06T03:16:17.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 --2- 172.21.15.153:0/2351403193 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d01085f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 --2- 172.21.15.153:0/2351403193 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 0x7f50d006fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- 172.21.15.153:0/2351403193 >> 172.21.15.153:0/2351403193 conn(0x7f50d00f7e50 msgr2=0x7f50d00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- 172.21.15.153:0/2351403193 shutdown_connections 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- 172.21.15.153:0/2351403193 wait complete. 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 Processor -- start 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- start start 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 0x7f50d0115580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d0116110 con 0x7f50d0070120 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.787+0000 7f50d77cb640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50d010f520 con 0x7f50d006f810 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:17.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45514/0 (socket says 172.21.15.153:45514) 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 -- 172.21.15.153:0/1982374516 learned_addr learned my addr 172.21.15.153:0/1982374516 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d5567640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 0x7f50d0115580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 -- 172.21.15.153:0/1982374516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 msgr2=0x7f50d0115580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 0x7f50d0115580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 -- 172.21.15.153:0/1982374516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c4005040 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d4d66640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f50c4000c00 tx=0x7f50c4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c4038070 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f50c40098b0 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50d010f7a0 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f50c403c770 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.788+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50d010fc00 con 0x7f50d0070120 2024-06-06T03:16:17.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.789+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50d0070f50 con 0x7f50d0070120 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.790+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f50c4030020 con 0x7f50d0070120 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.790+0000 7f50cdffb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 0x7f50b80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.790+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f50c40a22a0 con 0x7f50d0070120 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.790+0000 7f50d5567640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 0x7f50b80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.791+0000 7f50d5567640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 0x7f50b80678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50c8004c40 tx=0x7f50c8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:17.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.795+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f50c4070750 con 0x7f50d0070120 2024-06-06T03:16:17.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.996+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f50d00fbab0 con 0x7f50d0070120 2024-06-06T03:16:17.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.997+0000 7f50cdffb640 1 -- 172.21.15.153:0/1982374516 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f50c4073d70 con 0x7f50d0070120 2024-06-06T03:16:17.997 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:17.997 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 msgr2=0x7f50b80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 0x7f50b80678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f50c8004c40 tx=0x7f50c8009040 comp rx=0 tx=0).stop 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 msgr2=0x7f50d0115ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f50c4000c00 tx=0x7f50c4009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 shutdown_connections 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f50d0070120 0x7f50d0115ac0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f50b8065430 0x7f50b80678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:17.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 --2- 172.21.15.153:0/1982374516 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50d006f810 0x7f50d0115580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:18.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 >> 172.21.15.153:0/1982374516 conn(0x7f50d00f7e50 msgr2=0x7f50d00f9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:18.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 shutdown_connections 2024-06-06T03:16:18.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:17.999+0000 7f50d77cb640 1 -- 172.21.15.153:0/1982374516 wait complete. 2024-06-06T03:16:18.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:18 smithi173 ceph-mon[35650]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:18.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:18 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1982374516' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:18.638 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:18 smithi153 ceph-mon[29300]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:18.638 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:18 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1982374516' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:20.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:20 smithi173 ceph-mon[35650]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:20.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:20 smithi153 ceph-mon[29300]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:21.663 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:21.929 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:22.302 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:22 smithi153 ceph-mon[29300]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:22.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:22 smithi173 ceph-mon[35650]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:23.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 -- 172.21.15.153:0/3448496302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 msgr2=0x7f49c0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 --2- 172.21.15.153:0/3448496302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0102690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f49b80048d0 tx=0x7f49b8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 -- 172.21.15.153:0/3448496302 shutdown_connections 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 --2- 172.21.15.153:0/3448496302 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 --2- 172.21.15.153:0/3448496302 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0102690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 -- 172.21.15.153:0/3448496302 >> 172.21.15.153:0/3448496302 conn(0x7f49c00f7e50 msgr2=0x7f49c00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 -- 172.21.15.153:0/3448496302 shutdown_connections 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.581+0000 7f49c50f7640 1 -- 172.21.15.153:0/3448496302 wait complete. 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 Processor -- start 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 -- start start 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:23.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c0113e70 con 0x7f49c01022c0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.582+0000 7f49c50f7640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49c010d2b0 con 0x7f49c0102c60 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47356/0 (socket says 172.21.15.153:47356) 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 -- 172.21.15.153:0/106154556 learned_addr learned my addr 172.21.15.153:0/106154556 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49be59c640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 -- 172.21.15.153:0/106154556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 msgr2=0x7f49c0113870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c0113870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 -- 172.21.15.153:0/106154556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b8005040 con 0x7f49c01022c0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49be59c640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c0113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49bed9d640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f49b8003080 tx=0x7f49b8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b8038070 con 0x7f49c01022c0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f49b8008ee0 con 0x7f49c01022c0 2024-06-06T03:16:23.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49c010d530 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f49b803d870 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.583+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49c010da70 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.584+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49c00fda30 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.585+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f49b8030020 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.585+0000 7f49b77fe640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 0x7f49ac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.585+0000 7f49be59c640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 0x7f49ac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.585+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f49b80a2190 con 0x7f49c01022c0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.586+0000 7f49be59c640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 0x7f49ac0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f49a8007fd0 tx=0x7f49a8009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:23.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.589+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f49b8070760 con 0x7f49c01022c0 2024-06-06T03:16:23.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.789+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f49c004b480 con 0x7f49c01022c0 2024-06-06T03:16:23.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.789+0000 7f49b77fe640 1 -- 172.21.15.153:0/106154556 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f49b8073d80 con 0x7f49c01022c0 2024-06-06T03:16:23.789 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:23.789 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:23.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 msgr2=0x7f49ac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 0x7f49ac0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f49a8007fd0 tx=0x7f49a8009040 comp rx=0 tx=0).stop 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 msgr2=0x7f49c0113330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f49b8003080 tx=0x7f49b8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 shutdown_connections 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f49ac065430 0x7f49ac0678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f49c0102c60 0x7f49c0113870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 --2- 172.21.15.153:0/106154556 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f49c01022c0 0x7f49c0113330 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 >> 172.21.15.153:0/106154556 conn(0x7f49c00f7e50 msgr2=0x7f49c00f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 shutdown_connections 2024-06-06T03:16:23.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:23.791+0000 7f49c50f7640 1 -- 172.21.15.153:0/106154556 wait complete. 2024-06-06T03:16:24.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:24 smithi153 ceph-mon[29300]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:24.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:24 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/106154556' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:24.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:24 smithi173 ceph-mon[35650]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:24.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:24 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/106154556' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:26.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:26 smithi173 ceph-mon[35650]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:26.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:26.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:26 smithi153 ceph-mon[29300]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:26.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:27.392 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:27.658 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:28.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:28 smithi173 ceph-mon[35650]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:28.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:28 smithi153 ceph-mon[29300]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:29.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.296+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2050573944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb960 msgr2=0x7f0e241086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:29.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.296+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2050573944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb960 0x7f0e241086c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0e140048d0 tx=0x7f0e14009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2050573944 shutdown_connections 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2050573944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb960 0x7f0e241086c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2050573944 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb050 0x7f0e240fb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2050573944 >> 172.21.15.153:0/2050573944 conn(0x7f0e240f6dc0 msgr2=0x7f0e240f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2050573944 shutdown_connections 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2050573944 wait complete. 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 Processor -- start 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.297+0000 7f0e2c1ca640 1 -- start start 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb960 0x7f0e24115750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:29.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e24115d50 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2410f190 con 0x7f0e240fb960 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47382/0 (socket says 172.21.15.153:47382) 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 -- 172.21.15.153:0/2552760456 learned_addr learned my addr 172.21.15.153:0/2552760456 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 -- 172.21.15.153:0/2552760456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb960 msgr2=0x7f0e24115750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29765640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb960 0x7f0e24115750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb960 0x7f0e24115750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 -- 172.21.15.153:0/2552760456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e14005040 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e29f66640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e20002a50 tx=0x7f0e20004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e20007c30 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e2000f040 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2410f470 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e2000db50 con 0x7f0e240fb050 2024-06-06T03:16:29.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.298+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2410fa10 con 0x7f0e240fb050 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.299+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e240660b0 con 0x7f0e240fb050 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.300+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0e2001c400 con 0x7f0e240fb050 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.300+0000 7f0e1a7fc640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 0x7f0e100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.300+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0e20081e00 con 0x7f0e240fb050 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.300+0000 7f0e29765640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 0x7f0e100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.300+0000 7f0e29765640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 0x7f0e100678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e140043d0 tx=0x7f0e14009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:29.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.304+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0e200502b0 con 0x7f0e240fb050 2024-06-06T03:16:29.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.500+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0e2410b5f0 con 0x7f0e240fb050 2024-06-06T03:16:29.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.501+0000 7f0e1a7fc640 1 -- 172.21.15.153:0/2552760456 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0e200538d0 con 0x7f0e240fb050 2024-06-06T03:16:29.501 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:29.501 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 msgr2=0x7f0e100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 0x7f0e100678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0e140043d0 tx=0x7f0e14009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 msgr2=0x7f0e24115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f0e20002a50 tx=0x7f0e20004600 comp rx=0 tx=0).stop 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 shutdown_connections 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0e10065430 0x7f0e100678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0e240fb960 0x7f0e24115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 --2- 172.21.15.153:0/2552760456 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0e240fb050 0x7f0e24115210 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 >> 172.21.15.153:0/2552760456 conn(0x7f0e240f6dc0 msgr2=0x7f0e240f7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 shutdown_connections 2024-06-06T03:16:29.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:29.503+0000 7f0e2c1ca640 1 -- 172.21.15.153:0/2552760456 wait complete. 2024-06-06T03:16:30.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:30 smithi153 ceph-mon[29300]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:30.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:30 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2552760456' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:30.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:30 smithi173 ceph-mon[35650]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:30.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:30 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2552760456' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:32.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:32 smithi173 ceph-mon[35650]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:32.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:32 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:16:32.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:32 smithi153 ceph-mon[29300]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:32.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:32 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:16:33.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:33.453 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:16:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:16:33.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:16:33.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:16:33.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:16:33.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:16:34.571 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:34 smithi153 ceph-mon[29300]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:34.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:34 smithi173 ceph-mon[35650]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 -- 172.21.15.153:0/1713324592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 msgr2=0x7f69481000b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 --2- 172.21.15.153:0/1713324592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f69481000b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f693c0048d0 tx=0x7f693c009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 -- 172.21.15.153:0/1713324592 shutdown_connections 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 --2- 172.21.15.153:0/1713324592 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f69481000b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 --2- 172.21.15.153:0/1713324592 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69480fb4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.116+0000 7f694e902640 1 -- 172.21.15.153:0/1713324592 >> 172.21.15.153:0/1713324592 conn(0x7f69480f6d90 msgr2=0x7f69480f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:35.118 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 -- 172.21.15.153:0/1713324592 shutdown_connections 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 -- 172.21.15.153:0/1713324592 wait complete. 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 Processor -- start 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 -- start start 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69481196a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f694811a250 con 0x7f69480fba90 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.117+0000 7f694e902640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6948113640 con 0x7f69480fb0f0 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:46568/0 (socket says 172.21.15.153:46568) 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 -- 172.21.15.153:0/61748787 learned_addr learned my addr 172.21.15.153:0/61748787 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:35.119 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f6947fff640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69481196a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 -- 172.21.15.153:0/61748787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 msgr2=0x7f69481196a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69481196a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 -- 172.21.15.153:0/61748787 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693c005040 con 0x7f69480fba90 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f6947fff640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69481196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f69477fe640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f693c0043d0 tx=0x7f693c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f693c038070 con 0x7f69480fba90 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f693c032970 con 0x7f69480fba90 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69481138c0 con 0x7f69480fba90 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f693c03da60 con 0x7f69480fba90 2024-06-06T03:16:35.120 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.118+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6948113d80 con 0x7f69480fba90 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.119+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69480660d0 con 0x7f69480fba90 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.120+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f693c030020 con 0x7f69480fba90 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.120+0000 7f6944ff9640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 0x7f69300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.120+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f693c0a2630 con 0x7f69480fba90 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.120+0000 7f6947fff640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 0x7f69300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:35.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.121+0000 7f6947fff640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 0x7f69300678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6938009af0 tx=0x7f693800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:35.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.124+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f693c070ae0 con 0x7f69480fba90 2024-06-06T03:16:35.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.330+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f69480fa9d0 con 0x7f69480fba90 2024-06-06T03:16:35.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.330+0000 7f6944ff9640 1 -- 172.21.15.153:0/61748787 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f693c074100 con 0x7f69480fba90 2024-06-06T03:16:35.331 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:35.331 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 msgr2=0x7f69300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 0x7f69300678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6938009af0 tx=0x7f693800b040 comp rx=0 tx=0).stop 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 msgr2=0x7f6948119c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f693c0043d0 tx=0x7f693c009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 shutdown_connections 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69480fba90 0x7f6948119c00 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f6930065430 0x7f69300678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 --2- 172.21.15.153:0/61748787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f69480fb0f0 0x7f69481196a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:35.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 >> 172.21.15.153:0/61748787 conn(0x7f69480f6d90 msgr2=0x7f69480f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:35.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 shutdown_connections 2024-06-06T03:16:35.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:35.332+0000 7f694e902640 1 -- 172.21.15.153:0/61748787 wait complete. 2024-06-06T03:16:36.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:36 smithi173 ceph-mon[35650]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:36.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:36 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/61748787' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:36.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:36 smithi153 ceph-mon[29300]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:36.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:36 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/61748787' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:38.590 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:38 smithi173 ceph-mon[35650]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:38.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:38 smithi153 ceph-mon[29300]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:38.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:39.220 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:40.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:40 smithi173 ceph-mon[35650]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:40.606 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:40 smithi153 ceph-mon[29300]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- 172.21.15.153:0/621898658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 msgr2=0x7f97f80fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 --2- 172.21.15.153:0/621898658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f80fb940 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f97e80048d0 tx=0x7f97e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- 172.21.15.153:0/621898658 shutdown_connections 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 --2- 172.21.15.153:0/621898658 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 0x7f97f80fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 --2- 172.21.15.153:0/621898658 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f80fb940 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:40.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- 172.21.15.153:0/621898658 >> 172.21.15.153:0/621898658 conn(0x7f97f8072110 msgr2=0x7f97f8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- 172.21.15.153:0/621898658 shutdown_connections 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- 172.21.15.153:0/621898658 wait complete. 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 Processor -- start 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- start start 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 0x7f97f810e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f810ec80 con 0x7f97f80fb570 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.986+0000 7f9800615640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f810edc0 con 0x7f97f80fbe80 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33638/0 (socket says 172.21.15.153:33638) 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 -- 172.21.15.153:0/2780337546 learned_addr learned my addr 172.21.15.153:0/2780337546 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fdbb0640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 0x7f97f810e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:40.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 -- 172.21.15.153:0/2780337546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 msgr2=0x7f97f810e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 0x7f97f810e5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 -- 172.21.15.153:0/2780337546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e8005040 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97fe3b1640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f97e8003080 tx=0x7f97e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e8038070 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97e8008ee0 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f81170d0 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97e803d870 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.987+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f8117510 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.988+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f80fc920 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.989+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f97e8030020 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.989+0000 7f97eeffd640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 0x7f97e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.989+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f97e80a2250 con 0x7f97f80fb570 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.989+0000 7f97fdbb0640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 0x7f97e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:40.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.989+0000 7f97fdbb0640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 0x7f97e40678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97f8065dc0 tx=0x7f97f4008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:40.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:40.993+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f97e8070780 con 0x7f97f80fb570 2024-06-06T03:16:41.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.190+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f97f8072c20 con 0x7f97f80fb570 2024-06-06T03:16:41.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.190+0000 7f97eeffd640 1 -- 172.21.15.153:0/2780337546 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f97e8073da0 con 0x7f97f80fb570 2024-06-06T03:16:41.191 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:41.191 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 msgr2=0x7f97e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 0x7f97e40678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f97f8065dc0 tx=0x7f97f4008040 comp rx=0 tx=0).stop 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 msgr2=0x7f97f810e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f97e8003080 tx=0x7f97e8009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 shutdown_connections 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f97e4065430 0x7f97e40678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f97f80fbe80 0x7f97f810e5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 --2- 172.21.15.153:0/2780337546 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f97f80fb570 0x7f97f810e060 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:41.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 >> 172.21.15.153:0/2780337546 conn(0x7f97f8072110 msgr2=0x7f97f80f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:41.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 shutdown_connections 2024-06-06T03:16:41.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:41.192+0000 7f9800615640 1 -- 172.21.15.153:0/2780337546 wait complete. 2024-06-06T03:16:41.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:41.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:41 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2780337546' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:41.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:41.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:41 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2780337546' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:42.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:42 smithi173 ceph-mon[35650]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:42.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:42 smithi153 ceph-mon[29300]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:44.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:44 smithi173 ceph-mon[35650]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:44.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:44 smithi153 ceph-mon[29300]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:44.897 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:45.159 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:46.569 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:46 smithi153 ceph-mon[29300]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:46.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:46 smithi173 ceph-mon[35650]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 -- 172.21.15.153:0/3600745428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac01022c0 msgr2=0x7f1ac0102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/3600745428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac01022c0 0x7f1ac0102690 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00048d0 tx=0x7f1ab0009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 -- 172.21.15.153:0/3600745428 shutdown_connections 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/3600745428 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac0102c60 0x7f1ac00fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/3600745428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac01022c0 0x7f1ac0102690 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 -- 172.21.15.153:0/3600745428 >> 172.21.15.153:0/3600745428 conn(0x7f1ac00f7e50 msgr2=0x7f1ac00fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 -- 172.21.15.153:0/3600745428 shutdown_connections 2024-06-06T03:16:46.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.879+0000 7f1ac64ca640 1 -- 172.21.15.153:0/3600745428 wait complete. 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 Processor -- start 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 -- start start 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac01022c0 0x7f1ac0113330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0113e70 con 0x7f1ac0102c60 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1ac64ca640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac010d2b0 con 0x7f1ac01022c0 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1abf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1abf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:33668/0 (socket says 172.21.15.153:33668) 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1abf7fe640 1 -- 172.21.15.153:0/4149266948 learned_addr learned my addr 172.21.15.153:0/4149266948 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:46.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.880+0000 7f1abffff640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac01022c0 0x7f1ac0113330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abf7fe640 1 -- 172.21.15.153:0/4149266948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac01022c0 msgr2=0x7f1ac0113330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abf7fe640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac01022c0 0x7f1ac0113330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abf7fe640 1 -- 172.21.15.153:0/4149266948 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab0005040 con 0x7f1ac0102c60 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abf7fe640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400eb70 tx=0x7f1ab400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab4007c30 con 0x7f1ac0102c60 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1ab400f040 con 0x7f1ac0102c60 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac010d530 con 0x7f1ac0102c60 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1ab400cad0 con 0x7f1ac0102c60 2024-06-06T03:16:46.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.881+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac010dad0 con 0x7f1ac0102c60 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.882+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac00fda30 con 0x7f1ac0102c60 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.883+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1ab400cc50 con 0x7f1ac0102c60 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.883+0000 7f1abcff9640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 0x7f1aa80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.883+0000 7f1abffff640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 0x7f1aa80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.883+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1ab40820e0 con 0x7f1ac0102c60 2024-06-06T03:16:46.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.884+0000 7f1abffff640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 0x7f1aa80678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00048a0 tx=0x7f1ab0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:46.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:46.887+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1ab4050700 con 0x7f1ac0102c60 2024-06-06T03:16:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.095+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1ac0109420 con 0x7f1ac0102c60 2024-06-06T03:16:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.095+0000 7f1abcff9640 1 -- 172.21.15.153:0/4149266948 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1ab4053d20 con 0x7f1ac0102c60 2024-06-06T03:16:47.096 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:47.096 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 msgr2=0x7f1aa80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 0x7f1aa80678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1ab00048a0 tx=0x7f1ab0035040 comp rx=0 tx=0).stop 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 msgr2=0x7f1ac0113870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ab400eb70 tx=0x7f1ab400b040 comp rx=0 tx=0).stop 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 shutdown_connections 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1ac0102c60 0x7f1ac0113870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1aa8065430 0x7f1aa80678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 --2- 172.21.15.153:0/4149266948 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1ac01022c0 0x7f1ac0113330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 >> 172.21.15.153:0/4149266948 conn(0x7f1ac00f7e50 msgr2=0x7f1ac00f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 shutdown_connections 2024-06-06T03:16:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:47.097+0000 7f1ac64ca640 1 -- 172.21.15.153:0/4149266948 wait complete. 2024-06-06T03:16:47.690 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:47 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4149266948' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:47.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:47 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4149266948' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:48.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:48 smithi153 ceph-mon[29300]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:48.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:48 smithi173 ceph-mon[35650]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:50.717 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:50.745 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:50 smithi153 ceph-mon[29300]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:50.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:50 smithi173 ceph-mon[35650]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:50.984 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:52.413 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:52 smithi153 ceph-mon[29300]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 -- 172.21.15.153:0/2127779848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 msgr2=0x7fd7480fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 --2- 172.21.15.153:0/2127779848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd7480fcd80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0048d0 tx=0x7fd73c009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 -- 172.21.15.153:0/2127779848 shutdown_connections 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 --2- 172.21.15.153:0/2127779848 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd7480fcd80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 --2- 172.21.15.153:0/2127779848 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 0x7fd7480fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.766+0000 7fd74e773640 1 -- 172.21.15.153:0/2127779848 >> 172.21.15.153:0/2127779848 conn(0x7fd748072110 msgr2=0x7fd748072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 -- 172.21.15.153:0/2127779848 shutdown_connections 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 -- 172.21.15.153:0/2127779848 wait complete. 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 Processor -- start 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 -- start start 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 0x7fd74810e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74810eb50 con 0x7fd7480fbe80 2024-06-06T03:16:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.767+0000 7fd74e773640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74810ecc0 con 0x7fd7480fb570 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57778/0 (socket says 172.21.15.153:57778) 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 -- 172.21.15.153:0/588537216 learned_addr learned my addr 172.21.15.153:0/588537216 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd747fff640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 0x7fd74810e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 -- 172.21.15.153:0/588537216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 msgr2=0x7fd74810e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 0x7fd74810e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 -- 172.21.15.153:0/588537216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73c005040 con 0x7fd7480fbe80 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd7477fe640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0043d0 tx=0x7fd73c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd73c038070 con 0x7fd7480fbe80 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd73c032970 con 0x7fd7480fbe80 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd748116fa0 con 0x7fd7480fbe80 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd73c03da60 con 0x7fd7480fbe80 2024-06-06T03:16:52.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.768+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd748117410 con 0x7fd7480fbe80 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.769+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7480fc920 con 0x7fd7480fbe80 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.769+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd73c030020 con 0x7fd7480fbe80 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.770+0000 7fd744ff9640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 0x7fd7300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.770+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd73c0a26a0 con 0x7fd7480fbe80 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.770+0000 7fd747fff640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 0x7fd7300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.770+0000 7fd747fff640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 0x7fd7300678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd738007fd0 tx=0x7fd738009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:52.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.774+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd73c070b50 con 0x7fd7480fbe80 2024-06-06T03:16:52.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:52 smithi173 ceph-mon[35650]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:52.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.970+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd748072c20 con 0x7fd7480fbe80 2024-06-06T03:16:52.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.971+0000 7fd744ff9640 1 -- 172.21.15.153:0/588537216 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd73c074170 con 0x7fd7480fbe80 2024-06-06T03:16:52.970 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:52.970 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:52.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.972+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 msgr2=0x7fd7300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:52.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.972+0000 7fd74e773640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 0x7fd7300678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fd738007fd0 tx=0x7fd738009e90 comp rx=0 tx=0).stop 2024-06-06T03:16:52.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.972+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 msgr2=0x7fd74810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.972+0000 7fd74e773640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd73c0043d0 tx=0x7fd73c009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 shutdown_connections 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd7480fbe80 0x7fd74810e610 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd730065430 0x7fd7300678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 --2- 172.21.15.153:0/588537216 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd7480fb570 0x7fd74810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 >> 172.21.15.153:0/588537216 conn(0x7fd748072110 msgr2=0x7fd7480ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 shutdown_connections 2024-06-06T03:16:52.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:52.973+0000 7fd74e773640 1 -- 172.21.15.153:0/588537216 wait complete. 2024-06-06T03:16:53.644 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:53 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/588537216' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:53.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:53 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/588537216' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:54.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:54 smithi153 ceph-mon[29300]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:54.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:54 smithi173 ceph-mon[35650]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:56.670 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:16:56.698 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:56 smithi153 ceph-mon[29300]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:56.699 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:56.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:56 smithi173 ceph-mon[35650]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:56.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:16:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 -- 172.21.15.153:0/100450126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 msgr2=0x7f7fe80fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/100450126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe80fb200 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7fd40048d0 tx=0x7f7fd4009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 -- 172.21.15.153:0/100450126 shutdown_connections 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/100450126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe80fb200 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/100450126 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 0x7f7fe81015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 -- 172.21.15.153:0/100450126 >> 172.21.15.153:0/100450126 conn(0x7f7fe80f6dc0 msgr2=0x7f7fe80f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 -- 172.21.15.153:0/100450126 shutdown_connections 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.563+0000 7f7feeaa5640 1 -- 172.21.15.153:0/100450126 wait complete. 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 Processor -- start 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 -- start start 2024-06-06T03:16:58.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 0x7f7fe81130c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe8113c90 con 0x7f7fe8101b70 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7feeaa5640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe810d040 con 0x7f7fe81011d0 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7fe7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57794/0 (socket says 172.21.15.153:57794) 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.564+0000 7f7fe7fff640 1 -- 172.21.15.153:0/568457100 learned_addr learned my addr 172.21.15.153:0/568457100 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fec841640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 0x7f7fe81130c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe7fff640 1 -- 172.21.15.153:0/568457100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 msgr2=0x7f7fe81130c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe7fff640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 0x7f7fe81130c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe7fff640 1 -- 172.21.15.153:0/568457100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd4005040 con 0x7f7fe8101b70 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe7fff640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4003080 tx=0x7f7fd4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:58.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fd4038070 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7fd4008ee0 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe810d2c0 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7fd403d870 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.565+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe810d780 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.566+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe80fc940 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.567+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7fd4030020 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.567+0000 7f7fe57fa640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 0x7f7fd80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.567+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7fd40a2250 con 0x7f7fe8101b70 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.567+0000 7f7fec841640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 0x7f7fd80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.568+0000 7f7fec841640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 0x7f7fd80678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009950 tx=0x7f7fdc008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:16:58.606 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.571+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7fd4070780 con 0x7f7fe8101b70 2024-06-06T03:16:58.607 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:58 smithi153 ceph-mon[29300]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:58.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.770+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7fe81083c0 con 0x7f7fe8101b70 2024-06-06T03:16:58.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.770+0000 7f7fe57fa640 1 -- 172.21.15.153:0/568457100 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7fd4073da0 con 0x7f7fe8101b70 2024-06-06T03:16:58.770 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:16:58.770 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 msgr2=0x7f7fd80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 0x7f7fd80678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009950 tx=0x7f7fdc008040 comp rx=0 tx=0).stop 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 msgr2=0x7f7fe8113600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7fd4003080 tx=0x7f7fd4009da0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 shutdown_connections 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7fe8101b70 0x7f7fe8113600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7fd8065430 0x7f7fd80678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 --2- 172.21.15.153:0/568457100 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7fe81011d0 0x7f7fe81130c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 >> 172.21.15.153:0/568457100 conn(0x7f7fe80f6dc0 msgr2=0x7f7fe8065850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 shutdown_connections 2024-06-06T03:16:58.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:16:58.772+0000 7f7feeaa5640 1 -- 172.21.15.153:0/568457100 wait complete. 2024-06-06T03:16:58.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:58 smithi173 ceph-mon[35650]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:16:59.491 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:16:59 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/568457100' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:16:59.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:16:59 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/568457100' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:00.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:00 smithi153 ceph-mon[29300]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:00.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:00 smithi173 ceph-mon[35650]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:02.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:02.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:02 smithi153 ceph-mon[29300]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:02.782 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:02.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:02 smithi173 ceph-mon[35650]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.594+0000 7f598ca90640 1 -- 172.21.15.153:0/2097891619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 msgr2=0x7f59880fe310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.594+0000 7f598ca90640 1 --2- 172.21.15.153:0/2097891619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59880fe310 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f59700048d0 tx=0x7f5970009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- 172.21.15.153:0/2097891619 shutdown_connections 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 --2- 172.21.15.153:0/2097891619 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59880fe310 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 --2- 172.21.15.153:0/2097891619 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 0x7f5988065400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- 172.21.15.153:0/2097891619 >> 172.21.15.153:0/2097891619 conn(0x7f5988072110 msgr2=0x7f5988072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- 172.21.15.153:0/2097891619 shutdown_connections 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- 172.21.15.153:0/2097891619 wait complete. 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 Processor -- start 2024-06-06T03:17:04.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- start start 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 0x7f5988113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5988113bb0 con 0x7f59880fdea0 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.595+0000 7f598ca90640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598810d040 con 0x7f5988065030 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:58320/0 (socket says 172.21.15.153:58320) 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 -- 172.21.15.153:0/892317342 learned_addr learned my addr 172.21.15.153:0/892317342 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f598659c640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 0x7f5988113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 -- 172.21.15.153:0/892317342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 msgr2=0x7f5988113070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 0x7f5988113070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 -- 172.21.15.153:0/892317342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5970005040 con 0x7f59880fdea0 2024-06-06T03:17:04.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597dd9b640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59700043d0 tx=0x7f5970009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5970038070 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5970032970 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598810d2c0 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f597003da60 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.596+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598810d780 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.597+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59880fda40 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.598+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5970030020 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.598+0000 7f597f7fe640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 0x7f59740678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.598+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f59700a25c0 con 0x7f59880fdea0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.598+0000 7f598659c640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 0x7f59740678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:04.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.599+0000 7f598659c640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 0x7f59740678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5978004ed0 tx=0x7f597800b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:04.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.602+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5970070a70 con 0x7f59880fdea0 2024-06-06T03:17:04.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:04 smithi153 ceph-mon[29300]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:04.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.799+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f598810e6e0 con 0x7f59880fdea0 2024-06-06T03:17:04.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.800+0000 7f597f7fe640 1 -- 172.21.15.153:0/892317342 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5970074090 con 0x7f59880fdea0 2024-06-06T03:17:04.799 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:04.800 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:04.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 msgr2=0x7f59740678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 0x7f59740678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5978004ed0 tx=0x7f597800b040 comp rx=0 tx=0).stop 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 msgr2=0x7f59881135b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f59700043d0 tx=0x7f5970009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 shutdown_connections 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59880fdea0 0x7f59881135b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5974065430 0x7f59740678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 --2- 172.21.15.153:0/892317342 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5988065030 0x7f5988113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 >> 172.21.15.153:0/892317342 conn(0x7f5988072110 msgr2=0x7f5988105120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 shutdown_connections 2024-06-06T03:17:04.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:04.802+0000 7f598ca90640 1 -- 172.21.15.153:0/892317342 wait complete. 2024-06-06T03:17:04.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:04 smithi173 ceph-mon[35650]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:05.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:05 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/892317342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:05.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:05 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/892317342' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:06.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:06 smithi153 ceph-mon[29300]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:06.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:06 smithi173 ceph-mon[35650]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:07.653 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:07 smithi153 ceph-mon[29300]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:07.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:07 smithi173 ceph-mon[35650]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:08.522 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:08.801 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:10.018 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:09 smithi153 ceph-mon[29300]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:10.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:09 smithi173 ceph-mon[35650]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- 172.21.15.153:0/3702222674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c06fd50 msgr2=0x7fad5c0701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 --2- 172.21.15.153:0/3702222674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c06fd50 0x7fad5c0701c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fad4c0048d0 tx=0x7fad4c009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- 172.21.15.153:0/3702222674 shutdown_connections 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 --2- 172.21.15.153:0/3702222674 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c06fd50 0x7fad5c0701c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 --2- 172.21.15.153:0/3702222674 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c071440 0x7fad5c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- 172.21.15.153:0/3702222674 >> 172.21.15.153:0/3702222674 conn(0x7fad5c0f7f00 msgr2=0x7fad5c0fa310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- 172.21.15.153:0/3702222674 shutdown_connections 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- 172.21.15.153:0/3702222674 wait complete. 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 Processor -- start 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.532+0000 7fad62afc640 1 -- start start 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad62afc640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 0x7fad5c119690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad62afc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad62afc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c11a2d0 con 0x7fad5c071440 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad62afc640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5c113630 con 0x7fad5c06fd50 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53876/0 (socket says 172.21.15.153:53876) 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 -- 172.21.15.153:0/769365223 learned_addr learned my addr 172.21.15.153:0/769365223 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:10.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad60898640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 0x7fad5c119690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 -- 172.21.15.153:0/769365223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 msgr2=0x7fad5c119690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 0x7fad5c119690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 -- 172.21.15.153:0/769365223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad4c005040 con 0x7fad5c071440 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad60898640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 0x7fad5c119690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad53fff640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad4c003080 tx=0x7fad4c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad4c038070 con 0x7fad5c071440 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fad4c032e00 con 0x7fad5c071440 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fad4c03de80 con 0x7fad5c071440 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.533+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c1138b0 con 0x7fad5c071440 2024-06-06T03:17:10.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.534+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c113d70 con 0x7fad5c071440 2024-06-06T03:17:10.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.535+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c070f50 con 0x7fad5c071440 2024-06-06T03:17:10.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.535+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fad4c030020 con 0x7fad5c071440 2024-06-06T03:17:10.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.535+0000 7fad517fa640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 0x7fad440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:10.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.535+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fad4c0a3120 con 0x7fad5c071440 2024-06-06T03:17:10.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.535+0000 7fad60898640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 0x7fad440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:10.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.536+0000 7fad60898640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 0x7fad440678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad5c109250 tx=0x7fad58008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:10.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.540+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fad4c071700 con 0x7fad5c071440 2024-06-06T03:17:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.737+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fad5c0fbb00 con 0x7fad5c071440 2024-06-06T03:17:10.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.738+0000 7fad517fa640 1 -- 172.21.15.153:0/769365223 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fad4c074d20 con 0x7fad5c071440 2024-06-06T03:17:10.739 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:10.739 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 msgr2=0x7fad440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:10.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 0x7fad440678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fad5c109250 tx=0x7fad58008040 comp rx=0 tx=0).stop 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 msgr2=0x7fad5c119bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fad4c003080 tx=0x7fad4c009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 shutdown_connections 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fad5c071440 0x7fad5c119bf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fad44065430 0x7fad440678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 --2- 172.21.15.153:0/769365223 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fad5c06fd50 0x7fad5c119690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 >> 172.21.15.153:0/769365223 conn(0x7fad5c0f7f00 msgr2=0x7fad5c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 shutdown_connections 2024-06-06T03:17:10.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:10.740+0000 7fad62afc640 1 -- 172.21.15.153:0/769365223 wait complete. 2024-06-06T03:17:11.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:10 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/769365223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:11.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:11.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:10 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/769365223' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:11.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:12.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:11 smithi153 ceph-mon[29300]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:12.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:11 smithi173 ceph-mon[35650]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:14.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:13 smithi153 ceph-mon[29300]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:14.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:13 smithi173 ceph-mon[35650]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:14.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:14.782 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:16.227 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:15 smithi153 ceph-mon[29300]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:16.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:15 smithi173 ceph-mon[35650]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 -- 172.21.15.153:0/2053963338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b20102c60 msgr2=0x7f1b200fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 --2- 172.21.15.153:0/2053963338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b20102c60 0x7f1b200fc2f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1b100048d0 tx=0x7f1b10009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 -- 172.21.15.153:0/2053963338 shutdown_connections 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 --2- 172.21.15.153:0/2053963338 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b20102c60 0x7f1b200fc2f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 --2- 172.21.15.153:0/2053963338 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b201022c0 0x7f1b20102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.573+0000 7f1b2765c640 1 -- 172.21.15.153:0/2053963338 >> 172.21.15.153:0/2053963338 conn(0x7f1b200f7e50 msgr2=0x7f1b200fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 -- 172.21.15.153:0/2053963338 shutdown_connections 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 -- 172.21.15.153:0/2053963338 wait complete. 2024-06-06T03:17:16.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 Processor -- start 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 -- start start 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b20102c60 0x7f1b20113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b20114020 con 0x7f1b201022c0 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b2765c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2010d3a0 con 0x7f1b20102c60 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b253f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b253f8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53884/0 (socket says 172.21.15.153:53884) 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b253f8640 1 -- 172.21.15.153:0/3214866090 learned_addr learned my addr 172.21.15.153:0/3214866090 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.574+0000 7f1b24bf7640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b20102c60 0x7f1b20113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b253f8640 1 -- 172.21.15.153:0/3214866090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b20102c60 msgr2=0x7f1b20113990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:16.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b253f8640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b20102c60 0x7f1b20113990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b253f8640 1 -- 172.21.15.153:0/3214866090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b10005040 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b253f8640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b790 tx=0x7f1b1c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b1c00c700 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2010d680 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1b1c00cd30 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1b1c013370 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.575+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2010db70 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.576+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b200fda30 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.576+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1b1c012070 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.577+0000 7f1b15ffb640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 0x7f1b0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.577+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1b1c081cd0 con 0x7f1b201022c0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.577+0000 7f1b24bf7640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 0x7f1b0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.577+0000 7f1b24bf7640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 0x7f1b0c0678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009f80 tx=0x7f1b10035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:16.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.581+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1b1c050200 con 0x7f1b201022c0 2024-06-06T03:17:16.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.781+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1b2004b480 con 0x7f1b201022c0 2024-06-06T03:17:16.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.782+0000 7f1b15ffb640 1 -- 172.21.15.153:0/3214866090 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1b1c053820 con 0x7f1b201022c0 2024-06-06T03:17:16.782 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:16.782 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 msgr2=0x7f1b0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 0x7f1b0c0678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b10009f80 tx=0x7f1b10035040 comp rx=0 tx=0).stop 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 msgr2=0x7f1b20113450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f1b1c00b790 tx=0x7f1b1c0091d0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 shutdown_connections 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1b0c065430 0x7f1b0c0678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1b20102c60 0x7f1b20113990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 --2- 172.21.15.153:0/3214866090 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b201022c0 0x7f1b20113450 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 >> 172.21.15.153:0/3214866090 conn(0x7f1b200f7e50 msgr2=0x7f1b200f8de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 shutdown_connections 2024-06-06T03:17:16.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:16.784+0000 7f1b2765c640 1 -- 172.21.15.153:0/3214866090 wait complete. 2024-06-06T03:17:17.088 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:16 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3214866090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:17.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:16 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3214866090' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:18.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:17 smithi153 ceph-mon[29300]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:18.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:17 smithi173 ceph-mon[35650]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:20.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:19 smithi153 ceph-mon[29300]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:20.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:19 smithi173 ceph-mon[35650]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:20.515 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:20.778 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:22.201 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:21 smithi153 ceph-mon[29300]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:22.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:21 smithi173 ceph-mon[35650]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 -- 172.21.15.153:0/2517883181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 msgr2=0x7fdb440fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 --2- 172.21.15.153:0/2517883181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb440fc2f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c0048d0 tx=0x7fdb2c009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 -- 172.21.15.153:0/2517883181 shutdown_connections 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 --2- 172.21.15.153:0/2517883181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb440fc2f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 --2- 172.21.15.153:0/2517883181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 0x7fdb44102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 -- 172.21.15.153:0/2517883181 >> 172.21.15.153:0/2517883181 conn(0x7fdb440f7e50 msgr2=0x7fdb440fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 -- 172.21.15.153:0/2517883181 shutdown_connections 2024-06-06T03:17:22.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.542+0000 7fdb496ab640 1 -- 172.21.15.153:0/2517883181 wait complete. 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 Processor -- start 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 -- start start 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 0x7fdb4406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb440701f0 con 0x7fdb44102c60 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb496ab640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb44070360 con 0x7fdb441022c0 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb3bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53544/0 (socket says 172.21.15.153:53544) 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb3bfff640 1 -- 172.21.15.153:0/1969890239 learned_addr learned my addr 172.21.15.153:0/1969890239 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb42ffd640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 0x7fdb4406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb3bfff640 1 -- 172.21.15.153:0/1969890239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 msgr2=0x7fdb4406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.543+0000 7fdb3bfff640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 0x7fdb4406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb3bfff640 1 -- 172.21.15.153:0/1969890239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb2c005040 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb3bfff640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c003080 tx=0x7fdb2c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2c038070 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdb2c008ee0 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb44070590 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb2c03d870 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.544+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb441199d0 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.545+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb440fda30 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.545+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdb2c030020 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.546+0000 7fdb3b7fe640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 0x7fdb300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.546+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdb2c0a2150 con 0x7fdb44102c60 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.546+0000 7fdb42ffd640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 0x7fdb300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.546+0000 7fdb42ffd640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 0x7fdb300678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb34004b20 tx=0x7fdb34009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.550+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdb2c070720 con 0x7fdb44102c60 2024-06-06T03:17:22.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.747+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdb44109490 con 0x7fdb44102c60 2024-06-06T03:17:22.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.747+0000 7fdb3b7fe640 1 -- 172.21.15.153:0/1969890239 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdb2c073d40 con 0x7fdb44102c60 2024-06-06T03:17:22.747 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:22.747 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 msgr2=0x7fdb300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 0x7fdb300678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdb34004b20 tx=0x7fdb34009040 comp rx=0 tx=0).stop 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 msgr2=0x7fdb4406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fdb2c003080 tx=0x7fdb2c009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 shutdown_connections 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdb44102c60 0x7fdb4406fcb0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fdb30065430 0x7fdb300678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 --2- 172.21.15.153:0/1969890239 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fdb441022c0 0x7fdb4406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:22.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 >> 172.21.15.153:0/1969890239 conn(0x7fdb440f7e50 msgr2=0x7fdb440f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:22.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 shutdown_connections 2024-06-06T03:17:22.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:22.749+0000 7fdb496ab640 1 -- 172.21.15.153:0/1969890239 wait complete. 2024-06-06T03:17:23.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:22 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1969890239' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:23.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:22 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1969890239' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:24.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:23 smithi153 ceph-mon[29300]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:24.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:23 smithi173 ceph-mon[35650]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:26.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:25 smithi153 ceph-mon[29300]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:26.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:25 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:26.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:25 smithi173 ceph-mon[35650]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:25 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:26.514 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:26.782 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:28.202 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:27 smithi153 ceph-mon[29300]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:28.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:27 smithi173 ceph-mon[35650]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 -- 172.21.15.153:0/2577088787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe0102370 msgr2=0x7f8fe0102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 --2- 172.21.15.153:0/2577088787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe0102370 0x7f8fe0102740 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0048d0 tx=0x7f8fdc009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 -- 172.21.15.153:0/2577088787 shutdown_connections 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 --2- 172.21.15.153:0/2577088787 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe00fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 --2- 172.21.15.153:0/2577088787 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe0102370 0x7f8fe0102740 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 -- 172.21.15.153:0/2577088787 >> 172.21.15.153:0/2577088787 conn(0x7f8fe00709e0 msgr2=0x7f8fe0070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 -- 172.21.15.153:0/2577088787 shutdown_connections 2024-06-06T03:17:28.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.405+0000 7f8fe7647640 1 -- 172.21.15.153:0/2577088787 wait complete. 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 Processor -- start 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 -- start start 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0102370 0x7f8fe0115750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe0115d50 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe7647640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fe010f190 con 0x7f8fe0102370 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe53e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe53e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53552/0 (socket says 172.21.15.153:53552) 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe53e3640 1 -- 172.21.15.153:0/437917173 learned_addr learned my addr 172.21.15.153:0/437917173 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:28.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe4be2640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0102370 0x7f8fe0115750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe53e3640 1 -- 172.21.15.153:0/437917173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0102370 msgr2=0x7f8fe0115750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.406+0000 7f8fe53e3640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0102370 0x7f8fe0115750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fe53e3640 1 -- 172.21.15.153:0/437917173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fdc005040 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fe53e3640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0043d0 tx=0x7f8fdc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fdc038070 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8fdc032970 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fe010f410 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8fdc03da60 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.407+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fe010f950 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.408+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fe0065570 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.408+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8fdc030020 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.409+0000 7f8fd5ffb640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 0x7f8fcc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:28.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.409+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8fdc0a2540 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.409+0000 7f8fe4be2640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 0x7f8fcc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:28.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.409+0000 7f8fe4be2640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 0x7f8fcc0678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0110ad0 tx=0x7f8fd000b040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:28.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.413+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8fdc070a60 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.611+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8fe004b480 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.611+0000 7f8fd5ffb640 1 -- 172.21.15.153:0/437917173 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8fdc074080 con 0x7f8fe00fa9c0 2024-06-06T03:17:28.611 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:28.611 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:28.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 msgr2=0x7f8fcc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:28.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 0x7f8fcc0678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8fe0110ad0 tx=0x7f8fd000b040 comp rx=0 tx=0).stop 2024-06-06T03:17:28.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 msgr2=0x7f8fe0115210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0043d0 tx=0x7f8fdc009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 shutdown_connections 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8fcc065430 0x7f8fcc0678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8fe0102370 0x7f8fe0115750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 --2- 172.21.15.153:0/437917173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8fe00fa9c0 0x7f8fe0115210 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 >> 172.21.15.153:0/437917173 conn(0x7f8fe00709e0 msgr2=0x7f8fe00f9520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 shutdown_connections 2024-06-06T03:17:28.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:28.613+0000 7f8fe7647640 1 -- 172.21.15.153:0/437917173 wait complete. 2024-06-06T03:17:29.207 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:28 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/437917173' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:29.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:28 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/437917173' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:30.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:29 smithi153 ceph-mon[29300]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:30.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:29 smithi173 ceph-mon[35650]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:32.236 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:32.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:31 smithi153 ceph-mon[29300]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:32.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:31 smithi173 ceph-mon[35650]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:32.502 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:33.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:32 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:17:33.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:32 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:17:33.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:33 smithi153 ceph-mon[29300]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:33.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:17:33.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:17:33.990 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:17:34.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.226+0000 7faae20a8640 1 -- 172.21.15.153:0/3527410514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc101b40 msgr2=0x7faadc0fb1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:34.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.226+0000 7faae20a8640 1 --2- 172.21.15.153:0/3527410514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc101b40 0x7faadc0fb1d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7faac40048d0 tx=0x7faac4009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.226+0000 7faae20a8640 1 -- 172.21.15.153:0/3527410514 shutdown_connections 2024-06-06T03:17:34.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.226+0000 7faae20a8640 1 --2- 172.21.15.153:0/3527410514 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc101b40 0x7faadc0fb1d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.226+0000 7faae20a8640 1 --2- 172.21.15.153:0/3527410514 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc1011a0 0x7faadc101570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- 172.21.15.153:0/3527410514 >> 172.21.15.153:0/3527410514 conn(0x7faadc0f6d50 msgr2=0x7faadc0f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- 172.21.15.153:0/3527410514 shutdown_connections 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- 172.21.15.153:0/3527410514 wait complete. 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 Processor -- start 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- start start 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc101b40 0x7faadc113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc113b30 con 0x7faadc1011a0 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faae20a8640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faadc10cf70 con 0x7faadc101b40 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faadb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:34.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faadb7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54074/0 (socket says 172.21.15.153:54074) 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faadb7fe640 1 -- 172.21.15.153:0/883659095 learned_addr learned my addr 172.21.15.153:0/883659095 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.227+0000 7faad3fff640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc101b40 0x7faadc113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faadb7fe640 1 -- 172.21.15.153:0/883659095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc101b40 msgr2=0x7faadc113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faadb7fe640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc101b40 0x7faadc113530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faadb7fe640 1 -- 172.21.15.153:0/883659095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac4005040 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faadb7fe640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faacc002a50 tx=0x7faacc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc007c30 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faadc10d250 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faacc00f040 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faacc00db50 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.228+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc10d7f0 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.229+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faadc0fc910 con 0x7faadc1011a0 2024-06-06T03:17:34.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.229+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7faacc01c400 con 0x7faadc1011a0 2024-06-06T03:17:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.230+0000 7faad8ff9640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 0x7faac80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.230+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faacc081e00 con 0x7faadc1011a0 2024-06-06T03:17:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.230+0000 7faad3fff640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 0x7faac80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.230+0000 7faad3fff640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 0x7faac80678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faadc10e8b0 tx=0x7faac4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:34.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.234+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faacc0502b0 con 0x7faadc1011a0 2024-06-06T03:17:34.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:33 smithi173 ceph-mon[35650]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:17:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:17:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:17:34.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.432+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7faadc04b480 con 0x7faadc1011a0 2024-06-06T03:17:34.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.432+0000 7faad8ff9640 1 -- 172.21.15.153:0/883659095 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7faacc0538d0 con 0x7faadc1011a0 2024-06-06T03:17:34.432 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:34.432 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 msgr2=0x7faac80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 0x7faac80678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7faadc10e8b0 tx=0x7faac4035040 comp rx=0 tx=0).stop 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 msgr2=0x7faadc112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7faacc002a50 tx=0x7faacc004600 comp rx=0 tx=0).stop 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 shutdown_connections 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7faac8065430 0x7faac80678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7faadc101b40 0x7faadc113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 --2- 172.21.15.153:0/883659095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7faadc1011a0 0x7faadc112ff0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:34.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.434+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 >> 172.21.15.153:0/883659095 conn(0x7faadc0f6d50 msgr2=0x7faadc065860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:34.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.435+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 shutdown_connections 2024-06-06T03:17:34.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:34.435+0000 7faae20a8640 1 -- 172.21.15.153:0/883659095 wait complete. 2024-06-06T03:17:35.062 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:34 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/883659095' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:35.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:34 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/883659095' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:36.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:35 smithi153 ceph-mon[29300]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:36.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:35 smithi173 ceph-mon[35650]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:38.089 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:38.210 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:37 smithi153 ceph-mon[29300]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:38.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:37 smithi173 ceph-mon[35650]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:38.348 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:39.919 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:39 smithi153 ceph-mon[29300]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 -- 172.21.15.153:0/799873929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c102c60 msgr2=0x7f0f0c0fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:40.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/799873929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c102c60 0x7f0f0c0fc2f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0f000048d0 tx=0x7f0f00009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 -- 172.21.15.153:0/799873929 shutdown_connections 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/799873929 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c102c60 0x7f0f0c0fc2f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/799873929 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 -- 172.21.15.153:0/799873929 >> 172.21.15.153:0/799873929 conn(0x7f0f0c0f7e50 msgr2=0x7f0f0c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 -- 172.21.15.153:0/799873929 shutdown_connections 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.230+0000 7f0f10c0f640 1 -- 172.21.15.153:0/799873929 wait complete. 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 Processor -- start 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 -- start start 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c102c60 0x7f0f0c113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c113e90 con 0x7f0f0c1022c0 2024-06-06T03:17:40.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f10c0f640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c10d2d0 con 0x7f0f0c102c60 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f0a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f0a59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44634/0 (socket says 172.21.15.153:44634) 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.231+0000 7f0f0a59c640 1 -- 172.21.15.153:0/652448918 learned_addr learned my addr 172.21.15.153:0/652448918 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f09d9b640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c102c60 0x7f0f0c113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f0a59c640 1 -- 172.21.15.153:0/652448918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c102c60 msgr2=0x7f0f0c113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f0a59c640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c102c60 0x7f0f0c113890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f0a59c640 1 -- 172.21.15.153:0/652448918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f00005040 con 0x7f0f0c1022c0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f0a59c640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00c830 tx=0x7f0efc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0efc00a850 con 0x7f0f0c1022c0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0efc00ae80 con 0x7f0f0c1022c0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c10d5b0 con 0x7f0f0c1022c0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0efc012ea0 con 0x7f0f0c1022c0 2024-06-06T03:17:40.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.232+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c10db50 con 0x7f0f0c1022c0 2024-06-06T03:17:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.233+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c0fda30 con 0x7f0f0c1022c0 2024-06-06T03:17:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.234+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0efc012400 con 0x7f0f0c1022c0 2024-06-06T03:17:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.234+0000 7f0efaffd640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 0x7f0ef40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:40.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.234+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0efc082620 con 0x7f0f0c1022c0 2024-06-06T03:17:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.234+0000 7f0f09d9b640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 0x7f0ef40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:40.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.234+0000 7f0f09d9b640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 0x7f0ef40678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0f00000c00 tx=0x7f0f00009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:40.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.238+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0efc050cb0 con 0x7f0f0c1022c0 2024-06-06T03:17:40.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:39 smithi173 ceph-mon[35650]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:40.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.434+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0f0c0fbab0 con 0x7f0f0c1022c0 2024-06-06T03:17:40.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.435+0000 7f0efaffd640 1 -- 172.21.15.153:0/652448918 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0efc0126b0 con 0x7f0f0c1022c0 2024-06-06T03:17:40.435 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:40.435 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:40.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 msgr2=0x7f0ef40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:40.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 0x7f0ef40678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0f00000c00 tx=0x7f0f00009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 msgr2=0x7f0f0c113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0efc00c830 tx=0x7f0efc004430 comp rx=0 tx=0).stop 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 shutdown_connections 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f0ef4065430 0x7f0ef40678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f0f0c102c60 0x7f0f0c113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 --2- 172.21.15.153:0/652448918 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0f0c1022c0 0x7f0f0c113350 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 >> 172.21.15.153:0/652448918 conn(0x7f0f0c0f7e50 msgr2=0x7f0f0c0f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 shutdown_connections 2024-06-06T03:17:40.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:40.437+0000 7f0f10c0f640 1 -- 172.21.15.153:0/652448918 wait complete. 2024-06-06T03:17:41.250 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:40 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/652448918' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:41.250 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:40 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:41.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:40 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/652448918' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:41.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:40 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:42.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:41 smithi153 ceph-mon[29300]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:42.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:41 smithi173 ceph-mon[35650]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:44.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:43 smithi153 ceph-mon[29300]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:44.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:44.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:43 smithi173 ceph-mon[35650]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:44.539 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:46.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:45 smithi153 ceph-mon[29300]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.228+0000 7f73d0749640 1 -- 172.21.15.153:0/1844488613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fe1c0 msgr2=0x7f73c80fe590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.228+0000 7f73d0749640 1 --2- 172.21.15.153:0/1844488613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fe1c0 0x7f73c80fe590 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f73c00048d0 tx=0x7f73c0009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- 172.21.15.153:0/1844488613 shutdown_connections 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 --2- 172.21.15.153:0/1844488613 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fa940 0x7f73c80fad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 --2- 172.21.15.153:0/1844488613 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fe1c0 0x7f73c80fe590 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- 172.21.15.153:0/1844488613 >> 172.21.15.153:0/1844488613 conn(0x7f73c8071860 msgr2=0x7f73c8071c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- 172.21.15.153:0/1844488613 shutdown_connections 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- 172.21.15.153:0/1844488613 wait complete. 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 Processor -- start 2024-06-06T03:17:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- start start 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fe1c0 0x7f73c8113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c8113b20 con 0x7f73c80fa940 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73d0749640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73c810cf60 con 0x7f73c80fe1c0 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73ce4e5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73ce4e5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:44652/0 (socket says 172.21.15.153:44652) 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73ce4e5640 1 -- 172.21.15.153:0/1955256071 learned_addr learned my addr 172.21.15.153:0/1955256071 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.229+0000 7f73cdce4640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fe1c0 0x7f73c8113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73ce4e5640 1 -- 172.21.15.153:0/1955256071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fe1c0 msgr2=0x7f73c8113520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73ce4e5640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fe1c0 0x7f73c8113520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73ce4e5640 1 -- 172.21.15.153:0/1955256071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c0005040 con 0x7f73c80fa940 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73ce4e5640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f73c00043d0 tx=0x7f73c0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c0038070 con 0x7f73c80fa940 2024-06-06T03:17:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c810d1e0 con 0x7f73c80fa940 2024-06-06T03:17:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f73c0032970 con 0x7f73c80fa940 2024-06-06T03:17:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f73c003da60 con 0x7f73c80fa940 2024-06-06T03:17:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.230+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c810d720 con 0x7f73c80fa940 2024-06-06T03:17:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.231+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8072370 con 0x7f73c80fa940 2024-06-06T03:17:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.232+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f73c0030020 con 0x7f73c80fa940 2024-06-06T03:17:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.232+0000 7f73c6ffd640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 0x7f73b00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.232+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f73c00a25c0 con 0x7f73c80fa940 2024-06-06T03:17:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.232+0000 7f73cdce4640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 0x7f73b00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.232+0000 7f73cdce4640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 0x7f73b00678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f73c810e8a0 tx=0x7f73bc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.236+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f73c0070a70 con 0x7f73c80fa940 2024-06-06T03:17:46.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:45 smithi173 ceph-mon[35650]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:46.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.433+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f73c8064ab0 con 0x7f73c80fa940 2024-06-06T03:17:46.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.434+0000 7f73c6ffd640 1 -- 172.21.15.153:0/1955256071 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f73c0074090 con 0x7f73c80fa940 2024-06-06T03:17:46.434 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:46.434 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:46.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.435+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 msgr2=0x7f73b00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:46.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.435+0000 7f73d0749640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 0x7f73b00678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f73c810e8a0 tx=0x7f73bc009040 comp rx=0 tx=0).stop 2024-06-06T03:17:46.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.435+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 msgr2=0x7f73c8112fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:46.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.435+0000 7f73d0749640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f73c00043d0 tx=0x7f73c0009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.435+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 shutdown_connections 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f73b0065430 0x7f73b00678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f73c80fe1c0 0x7f73c8113520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 --2- 172.21.15.153:0/1955256071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f73c80fa940 0x7f73c8112fe0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 >> 172.21.15.153:0/1955256071 conn(0x7f73c8071860 msgr2=0x7f73c80f9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 shutdown_connections 2024-06-06T03:17:46.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:46.436+0000 7f73d0749640 1 -- 172.21.15.153:0/1955256071 wait complete. 2024-06-06T03:17:47.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:46 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1955256071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:47.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:46 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1955256071' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:48.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:47 smithi153 ceph-mon[29300]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:48.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:47 smithi173 ceph-mon[35650]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:50.058 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:50.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:49 smithi153 ceph-mon[29300]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:50.322 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:50.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:49 smithi173 ceph-mon[35650]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:52.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 -- 172.21.15.153:0/1619376598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc1022a0 msgr2=0x7f39bc102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:52.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/1619376598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc1022a0 0x7f39bc102670 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f39ac0048d0 tx=0x7f39ac009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 -- 172.21.15.153:0/1619376598 shutdown_connections 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/1619376598 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc102c40 0x7f39bc0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/1619376598 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc1022a0 0x7f39bc102670 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 -- 172.21.15.153:0/1619376598 >> 172.21.15.153:0/1619376598 conn(0x7f39bc0f7e70 msgr2=0x7f39bc0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 -- 172.21.15.153:0/1619376598 shutdown_connections 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 -- 172.21.15.153:0/1619376598 wait complete. 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.078+0000 7f39c0cc4640 1 Processor -- start 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39c0cc4640 1 -- start start 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39c0cc4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 0x7f39bc06f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39c0cc4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:52.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39c0cc4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0701f0 con 0x7f39bc102c40 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39c0cc4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc070360 con 0x7f39bc1022a0 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60678/0 (socket says 172.21.15.153:60678) 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 -- 172.21.15.153:0/3812273665 learned_addr learned my addr 172.21.15.153:0/3812273665 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39ba59c640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 0x7f39bc06f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 -- 172.21.15.153:0/3812273665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 msgr2=0x7f39bc06f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 0x7f39bc06f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 -- 172.21.15.153:0/3812273665 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac005040 con 0x7f39bc102c40 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39ba59c640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 0x7f39bc06f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b9d9b640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f39a4002a50 tx=0x7f39a4004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.079+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a4007c30 con 0x7f39bc102c40 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.080+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc0705f0 con 0x7f39bc102c40 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.080+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f39a400f040 con 0x7f39bc102c40 2024-06-06T03:17:52.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.080+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39a400db50 con 0x7f39bc102c40 2024-06-06T03:17:52.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.080+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc119720 con 0x7f39bc102c40 2024-06-06T03:17:52.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.081+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc0fda10 con 0x7f39bc102c40 2024-06-06T03:17:52.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.081+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f39a401c400 con 0x7f39bc102c40 2024-06-06T03:17:52.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.081+0000 7f39b2ffd640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 0x7f39a80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:52.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.081+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f39a4081e00 con 0x7f39bc102c40 2024-06-06T03:17:52.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.081+0000 7f39ba59c640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 0x7f39a80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:52.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.082+0000 7f39ba59c640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 0x7f39a80678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39ac032430 tx=0x7f39ac035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:52.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.086+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f39a40502b0 con 0x7f39bc102c40 2024-06-06T03:17:52.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:51 smithi153 ceph-mon[29300]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:52.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.284+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f39bc109460 con 0x7f39bc102c40 2024-06-06T03:17:52.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.284+0000 7f39b2ffd640 1 -- 172.21.15.153:0/3812273665 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f39a40538d0 con 0x7f39bc102c40 2024-06-06T03:17:52.285 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:52.285 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 msgr2=0x7f39a80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 0x7f39a80678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f39ac032430 tx=0x7f39ac035040 comp rx=0 tx=0).stop 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 msgr2=0x7f39bc06fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f39a4002a50 tx=0x7f39a4004600 comp rx=0 tx=0).stop 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 shutdown_connections 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39bc102c40 0x7f39bc06fcb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39a8065430 0x7f39a80678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 --2- 172.21.15.153:0/3812273665 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f39bc1022a0 0x7f39bc06f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:52.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 >> 172.21.15.153:0/3812273665 conn(0x7f39bc0f7e70 msgr2=0x7f39bc0f8e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:52.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.286+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 shutdown_connections 2024-06-06T03:17:52.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:52.287+0000 7f39c0cc4640 1 -- 172.21.15.153:0/3812273665 wait complete. 2024-06-06T03:17:52.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:51 smithi173 ceph-mon[35650]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:53.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:52 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3812273665' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:53.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:52 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3812273665' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:54.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:53 smithi153 ceph-mon[29300]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:54.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:53 smithi173 ceph-mon[35650]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:55.922 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:17:56.187 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:17:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:55 smithi153 ceph-mon[29300]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:56.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:55 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:56.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:55 smithi173 ceph-mon[35650]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:56.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:55 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:17:57.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- 172.21.15.153:0/400779630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd418102c60 msgr2=0x7fd4180fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:57.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 --2- 172.21.15.153:0/400779630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd418102c60 0x7fd4180fc2f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd40c0048d0 tx=0x7fd40c009da0 comp rx=0 tx=0).stop 2024-06-06T03:17:57.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- 172.21.15.153:0/400779630 shutdown_connections 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 --2- 172.21.15.153:0/400779630 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd418102c60 0x7fd4180fc2f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 --2- 172.21.15.153:0/400779630 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd4181022c0 0x7fd418102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- 172.21.15.153:0/400779630 >> 172.21.15.153:0/400779630 conn(0x7fd4180f7e50 msgr2=0x7fd4180fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- 172.21.15.153:0/400779630 shutdown_connections 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- 172.21.15.153:0/400779630 wait complete. 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 Processor -- start 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- start start 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 0x7fd418113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd418113e90 con 0x7fd4181022c0 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.898+0000 7fd41ff18640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41810d2d0 con 0x7fd418102c60 2024-06-06T03:17:57.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:60698/0 (socket says 172.21.15.153:60698) 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 -- 172.21.15.153:0/1454554231 learned_addr learned my addr 172.21.15.153:0/1454554231 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41d4b3640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 0x7fd418113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 -- 172.21.15.153:0/1454554231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 msgr2=0x7fd418113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 0x7fd418113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 -- 172.21.15.153:0/1454554231 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c005040 con 0x7fd4181022c0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41d4b3640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 0x7fd418113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41dcb4640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c810 tx=0x7fd4140043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd41400a820 con 0x7fd4181022c0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41810d5b0 con 0x7fd4181022c0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd41400ae50 con 0x7fd4181022c0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd4140115c0 con 0x7fd4181022c0 2024-06-06T03:17:57.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.899+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41810db50 con 0x7fd4181022c0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.900+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4180fda30 con 0x7fd4181022c0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.901+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd41400a980 con 0x7fd4181022c0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.901+0000 7fd40a7fc640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 0x7fd4040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.901+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd414082a60 con 0x7fd4181022c0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.901+0000 7fd41d4b3640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 0x7fd4040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.902+0000 7fd41d4b3640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 0x7fd4040678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd41810ec10 tx=0x7fd40c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:17:57.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:57.905+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd414050f10 con 0x7fd4181022c0 2024-06-06T03:17:58.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:57 smithi153 ceph-mon[29300]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:58.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.106+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd418109400 con 0x7fd4181022c0 2024-06-06T03:17:58.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.106+0000 7fd40a7fc640 1 -- 172.21.15.153:0/1454554231 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd414054530 con 0x7fd4181022c0 2024-06-06T03:17:58.106 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:17:58.106 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:17:58.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 msgr2=0x7fd4040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 0x7fd4040678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd41810ec10 tx=0x7fd40c035040 comp rx=0 tx=0).stop 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 msgr2=0x7fd418113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd41400c810 tx=0x7fd4140043c0 comp rx=0 tx=0).stop 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 shutdown_connections 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd404065430 0x7fd4040678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd418102c60 0x7fd418113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 --2- 172.21.15.153:0/1454554231 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd4181022c0 0x7fd418113350 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 >> 172.21.15.153:0/1454554231 conn(0x7fd4180f7e50 msgr2=0x7fd4180f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 shutdown_connections 2024-06-06T03:17:58.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:17:58.108+0000 7fd41ff18640 1 -- 172.21.15.153:0/1454554231 wait complete. 2024-06-06T03:17:58.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:57 smithi173 ceph-mon[35650]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:17:59.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:58 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1454554231' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:17:59.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:58 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1454554231' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:00.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:17:59 smithi153 ceph-mon[29300]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:00.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:17:59 smithi173 ceph-mon[35650]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:01.803 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:02.070 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:02.192 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:01 smithi153 ceph-mon[29300]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:02.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:01 smithi173 ceph-mon[35650]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:03.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 -- 172.21.15.153:0/1296727568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 msgr2=0x7f06b80fc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 --2- 172.21.15.153:0/1296727568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b80fc4d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0048d0 tx=0x7f06ac009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 -- 172.21.15.153:0/1296727568 shutdown_connections 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 --2- 172.21.15.153:0/1296727568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b8100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 --2- 172.21.15.153:0/1296727568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b80fc4d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 -- 172.21.15.153:0/1296727568 >> 172.21.15.153:0/1296727568 conn(0x7f06b80724a0 msgr2=0x7f06b80728a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 -- 172.21.15.153:0/1296727568 shutdown_connections 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.735+0000 7f06bf05a640 1 -- 172.21.15.153:0/1296727568 wait complete. 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 Processor -- start 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 -- start start 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b81124e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b8112ae0 con 0x7f06b80fc100 2024-06-06T03:18:03.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bf05a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b810bf20 con 0x7f06b80fcaa0 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bcdf6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bcdf6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47424/0 (socket says 172.21.15.153:47424) 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.736+0000 7f06bcdf6640 1 -- 172.21.15.153:0/752027611 learned_addr learned my addr 172.21.15.153:0/752027611 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06b7fff640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b81124e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bcdf6640 1 -- 172.21.15.153:0/752027611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 msgr2=0x7f06b81124e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bcdf6640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b81124e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bcdf6640 1 -- 172.21.15.153:0/752027611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06ac005040 con 0x7f06b80fc100 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06b7fff640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b81124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bcdf6640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0043d0 tx=0x7f06ac009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac038070 con 0x7f06b80fc100 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b810c1a0 con 0x7f06b80fc100 2024-06-06T03:18:03.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f06ac032970 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f06ac03da60 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.737+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b810c660 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.738+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b80fa800 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.739+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f06ac030020 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.739+0000 7f06b57fa640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 0x7f06a80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.739+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f06ac0a26d0 con 0x7f06b80fc100 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.739+0000 7f06b7fff640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 0x7f06a80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.739+0000 7f06b7fff640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 0x7f06a80678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f06a4007fd0 tx=0x7f06a4009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:03.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.743+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f06ac070b80 con 0x7f06b80fc100 2024-06-06T03:18:03.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.941+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f06b804b480 con 0x7f06b80fc100 2024-06-06T03:18:03.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.942+0000 7f06b57fa640 1 -- 172.21.15.153:0/752027611 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f06ac0741a0 con 0x7f06b80fc100 2024-06-06T03:18:03.942 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:03.942 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:03.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 msgr2=0x7f06a80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:03.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 0x7f06a80678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f06a4007fd0 tx=0x7f06a4009e90 comp rx=0 tx=0).stop 2024-06-06T03:18:03.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 msgr2=0x7f06b8111fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:03.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f06ac0043d0 tx=0x7f06ac009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 shutdown_connections 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f06a8065430 0x7f06a80678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f06b80fcaa0 0x7f06b81124e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 --2- 172.21.15.153:0/752027611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06b80fc100 0x7f06b8111fa0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 >> 172.21.15.153:0/752027611 conn(0x7f06b80724a0 msgr2=0x7f06b80f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 shutdown_connections 2024-06-06T03:18:03.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:03.944+0000 7f06bf05a640 1 -- 172.21.15.153:0/752027611 wait complete. 2024-06-06T03:18:04.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:03 smithi153 ceph-mon[29300]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:04.253 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:03 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/752027611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:04.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:03 smithi173 ceph-mon[35650]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:04.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:03 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/752027611' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:06.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:05 smithi153 ceph-mon[29300]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:06.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:05 smithi173 ceph-mon[35650]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:07.664 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:07.927 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:08.213 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:07 smithi153 ceph-mon[29300]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:08.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:07 smithi173 ceph-mon[35650]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 -- 172.21.15.153:0/1740176411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101cb0 msgr2=0x7f65640fb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 --2- 172.21.15.153:0/1740176411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101cb0 0x7f65640fb230 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f65580048d0 tx=0x7f6558009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 -- 172.21.15.153:0/1740176411 shutdown_connections 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 --2- 172.21.15.153:0/1740176411 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101cb0 0x7f65640fb230 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 --2- 172.21.15.153:0/1740176411 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101310 0x7f65641016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:09.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 -- 172.21.15.153:0/1740176411 >> 172.21.15.153:0/1740176411 conn(0x7f65640f6d90 msgr2=0x7f65640f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 -- 172.21.15.153:0/1740176411 shutdown_connections 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.816+0000 7f65695e1640 1 -- 172.21.15.153:0/1740176411 wait complete. 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 Processor -- start 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 -- start start 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 0x7f6564113640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6564113c40 con 0x7f6564101310 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65695e1640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656410d080 con 0x7f6564101cb0 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f6562ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47440/0 (socket says 172.21.15.153:47440) 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f6562ffd640 1 -- 172.21.15.153:0/282130073 learned_addr learned my addr 172.21.15.153:0/282130073 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:09.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.817+0000 7f65627fc640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 0x7f6564113640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f6562ffd640 1 -- 172.21.15.153:0/282130073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 msgr2=0x7f6564113640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f6562ffd640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 0x7f6564113640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f6562ffd640 1 -- 172.21.15.153:0/282130073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6558005040 con 0x7f6564101310 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f65627fc640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 0x7f6564113640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f6562ffd640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f655400c830 tx=0x7f6554004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f655400a850 con 0x7f6564101310 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f65695e1640 1 -- 172.21.15.153:0/282130073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656410d360 con 0x7f6564101310 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f655400ae80 con 0x7f6564101310 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6554012ea0 con 0x7f6564101310 2024-06-06T03:18:09.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.818+0000 7f65695e1640 1 -- 172.21.15.153:0/282130073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f656410d900 con 0x7f6564101310 2024-06-06T03:18:09.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.819+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65640fa9f0 con 0x7f6564101310 2024-06-06T03:18:09.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.820+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6554012400 con 0x7f6564101310 2024-06-06T03:18:09.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.820+0000 7f654b7fe640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 0x7f654c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:09.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.820+0000 7f65627fc640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 0x7f654c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:09.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.820+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6554082730 con 0x7f6564101310 2024-06-06T03:18:09.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.821+0000 7f65627fc640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 0x7f654c0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f656410e9c0 tx=0x7f6558035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:09.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:09.824+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6554050c60 con 0x7f6564101310 2024-06-06T03:18:10.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:09 smithi153 ceph-mon[29300]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.030+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f656404b480 con 0x7f6564101310 2024-06-06T03:18:10.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.030+0000 7f654b7fe640 1 -- 172.21.15.153:0/282130073 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f65540126b0 con 0x7f6564101310 2024-06-06T03:18:10.029 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:10.030 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 msgr2=0x7f654c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 0x7f654c0678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f656410e9c0 tx=0x7f6558035040 comp rx=0 tx=0).stop 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 msgr2=0x7f6564113100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f655400c830 tx=0x7f6554004430 comp rx=0 tx=0).stop 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 shutdown_connections 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f654c065430 0x7f654c0678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f6564101cb0 0x7f6564113640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 --2- 172.21.15.153:0/282130073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6564101310 0x7f6564113100 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 >> 172.21.15.153:0/282130073 conn(0x7f65640f6d90 msgr2=0x7f65640f7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:10.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 shutdown_connections 2024-06-06T03:18:10.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:10.032+0000 7f65497fa640 1 -- 172.21.15.153:0/282130073 wait complete. 2024-06-06T03:18:10.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:09 smithi173 ceph-mon[35650]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:11.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:10 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/282130073' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:11.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:10 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:11.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:10 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/282130073' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:11.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:10 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:12.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:11 smithi153 ceph-mon[29300]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:12.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:11 smithi173 ceph-mon[35650]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:13.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:14.091 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:14.214 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:13 smithi153 ceph-mon[29300]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:14.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:13 smithi173 ceph-mon[35650]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 -- 172.21.15.153:0/3376153696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1394102c60 msgr2=0x7f13940fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 --2- 172.21.15.153:0/3376153696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1394102c60 0x7f13940fc2f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f13840048d0 tx=0x7f1384009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 -- 172.21.15.153:0/3376153696 shutdown_connections 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 --2- 172.21.15.153:0/3376153696 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1394102c60 0x7f13940fc2f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 --2- 172.21.15.153:0/3376153696 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f13941022c0 0x7f1394102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 -- 172.21.15.153:0/3376153696 >> 172.21.15.153:0/3376153696 conn(0x7f13940f7e50 msgr2=0x7f13940fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 -- 172.21.15.153:0/3376153696 shutdown_connections 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.795+0000 7f139bc58640 1 -- 172.21.15.153:0/3376153696 wait complete. 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 Processor -- start 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 -- start start 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:15.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1394102c60 0x7f1394113890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1394113e90 con 0x7f13941022c0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f139bc58640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139410d2d0 con 0x7f1394102c60 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57008/0 (socket says 172.21.15.153:57008) 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 -- 172.21.15.153:0/4105886581 learned_addr learned my addr 172.21.15.153:0/4105886581 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13991f3640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1394102c60 0x7f1394113890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 -- 172.21.15.153:0/4105886581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1394102c60 msgr2=0x7f1394113890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1394102c60 0x7f1394113890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 -- 172.21.15.153:0/4105886581 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1384005040 con 0x7f13941022c0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f13999f4640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1390002a50 tx=0x7f1390004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1390007c30 con 0x7f13941022c0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.796+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f139000f040 con 0x7f13941022c0 2024-06-06T03:18:15.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.797+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139410d5b0 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.797+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f139000db60 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.797+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139410db20 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.797+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13940fda30 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.798+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f139000dcc0 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.798+0000 7f138a7fc640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 0x7f13800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.798+0000 7f13991f3640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 0x7f13800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.798+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1390082240 con 0x7f13941022c0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.799+0000 7f13991f3640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 0x7f13800678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1384009f80 tx=0x7f1384035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:15.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:15.802+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1390050830 con 0x7f13941022c0 2024-06-06T03:18:16.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.001+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1394109400 con 0x7f13941022c0 2024-06-06T03:18:16.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.001+0000 7f138a7fc640 1 -- 172.21.15.153:0/4105886581 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1390053e50 con 0x7f13941022c0 2024-06-06T03:18:16.001 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:16.001 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:16.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 msgr2=0x7f13800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 0x7f13800678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1384009f80 tx=0x7f1384035040 comp rx=0 tx=0).stop 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 msgr2=0x7f1394113350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f1390002a50 tx=0x7f1390004600 comp rx=0 tx=0).stop 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 shutdown_connections 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f1380065430 0x7f13800678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f1394102c60 0x7f1394113890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 --2- 172.21.15.153:0/4105886581 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f13941022c0 0x7f1394113350 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 >> 172.21.15.153:0/4105886581 conn(0x7f13940f7e50 msgr2=0x7f13940f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 shutdown_connections 2024-06-06T03:18:16.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:16.003+0000 7f139bc58640 1 -- 172.21.15.153:0/4105886581 wait complete. 2024-06-06T03:18:16.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:15 smithi153 ceph-mon[29300]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:16.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:15 smithi173 ceph-mon[35650]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:17.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:16 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4105886581' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:17.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:16 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4105886581' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:18.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:17 smithi153 ceph-mon[29300]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:18.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:17 smithi173 ceph-mon[35650]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:19.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:19.946 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:20.230 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:19 smithi153 ceph-mon[29300]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:20.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:19 smithi173 ceph-mon[35650]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:21.667 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 -- 172.21.15.153:0/454326568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 msgr2=0x7f50580fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 --2- 172.21.15.153:0/454326568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f50580fc2d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f50400048d0 tx=0x7f5040009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 -- 172.21.15.153:0/454326568 shutdown_connections 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 --2- 172.21.15.153:0/454326568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f50580fc2d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 --2- 172.21.15.153:0/454326568 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 -- 172.21.15.153:0/454326568 >> 172.21.15.153:0/454326568 conn(0x7f50580f7e70 msgr2=0x7f50580fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 -- 172.21.15.153:0/454326568 shutdown_connections 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.661+0000 7f505d60c640 1 -- 172.21.15.153:0/454326568 wait complete. 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 Processor -- start 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 -- start start 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058113000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:21.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5058113b40 con 0x7f5058102c40 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f505d60c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810cf80 con 0x7f50581022a0 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57352/0 (socket says 172.21.15.153:57352) 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 -- 172.21.15.153:0/514001181 learned_addr learned my addr 172.21.15.153:0/514001181 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f5056ffd640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058113000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 -- 172.21.15.153:0/514001181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 msgr2=0x7f5058113000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058113000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f504e7fc640 1 -- 172.21.15.153:0/514001181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040005040 con 0x7f5058102c40 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.662+0000 7f5056ffd640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f504e7fc640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5040003080 tx=0x7f5040009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5040038070 con 0x7f5058102c40 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5040008ee0 con 0x7f5058102c40 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505810d200 con 0x7f5058102c40 2024-06-06T03:18:21.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f504003d870 con 0x7f5058102c40 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.663+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505810d740 con 0x7f5058102c40 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.664+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50580fda10 con 0x7f5058102c40 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.664+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5040030020 con 0x7f5058102c40 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.665+0000 7f504ffff640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 0x7f50440678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.665+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f50400a2150 con 0x7f5058102c40 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.665+0000 7f5056ffd640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 0x7f50440678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.665+0000 7f5056ffd640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 0x7f50440678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:21.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.669+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5040070720 con 0x7f5058102c40 2024-06-06T03:18:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.866+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f505810e670 con 0x7f5058102c40 2024-06-06T03:18:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.867+0000 7f504ffff640 1 -- 172.21.15.153:0/514001181 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5040073d40 con 0x7f5058102c40 2024-06-06T03:18:21.866 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:21.867 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 msgr2=0x7f50440678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 0x7f50440678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048009040 comp rx=0 tx=0).stop 2024-06-06T03:18:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 msgr2=0x7f5058113540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f5040003080 tx=0x7f5040009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 shutdown_connections 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5058102c40 0x7f5058113540 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5044065430 0x7f50440678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 --2- 172.21.15.153:0/514001181 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f50581022a0 0x7f5058113000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 >> 172.21.15.153:0/514001181 conn(0x7f50580f7e70 msgr2=0x7f50580f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 shutdown_connections 2024-06-06T03:18:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:21.869+0000 7f505d60c640 1 -- 172.21.15.153:0/514001181 wait complete. 2024-06-06T03:18:22.213 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:21 smithi153 ceph-mon[29300]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:22.213 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:21 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/514001181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:22.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:21 smithi173 ceph-mon[35650]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:22.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:21 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/514001181' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:24.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:23 smithi153 ceph-mon[29300]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:24.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:23 smithi173 ceph-mon[35650]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:25.598 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:25.875 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:26.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:25 smithi153 ceph-mon[29300]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:26.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:25 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:26.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:25 smithi173 ceph-mon[35650]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:25 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:27.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 -- 172.21.15.153:0/251123484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 msgr2=0x7f7b700fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:27.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 --2- 172.21.15.153:0/251123484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b700fb1e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7b580048d0 tx=0x7f7b58009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 -- 172.21.15.153:0/251123484 shutdown_connections 2024-06-06T03:18:27.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 --2- 172.21.15.153:0/251123484 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b700fb1e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 --2- 172.21.15.153:0/251123484 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 0x7f7b70101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 -- 172.21.15.153:0/251123484 >> 172.21.15.153:0/251123484 conn(0x7f7b700f6d60 msgr2=0x7f7b700f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 -- 172.21.15.153:0/251123484 shutdown_connections 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.524+0000 7f7b755bd640 1 -- 172.21.15.153:0/251123484 wait complete. 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 Processor -- start 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 -- start start 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 0x7f7b701130d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b70113ca0 con 0x7f7b70101b50 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b755bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7010d050 con 0x7f7b701011b0 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b667fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b667fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57370/0 (socket says 172.21.15.153:57370) 2024-06-06T03:18:27.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b667fc640 1 -- 172.21.15.153:0/1132472560 learned_addr learned my addr 172.21.15.153:0/1132472560 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.525+0000 7f7b6effd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 0x7f7b701130d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b667fc640 1 -- 172.21.15.153:0/1132472560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 msgr2=0x7f7b701130d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b667fc640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 0x7f7b701130d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b667fc640 1 -- 172.21.15.153:0/1132472560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b58005040 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b667fc640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7b58003080 tx=0x7f7b58009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b58038070 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7010d2d0 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7b58008ee0 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7b5803d870 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.526+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7010d790 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.527+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b700fc920 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.527+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7b58030020 con 0x7f7b70101b50 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.528+0000 7f7b67fff640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 0x7f7b5c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:27.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.528+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7b580a2250 con 0x7f7b70101b50 2024-06-06T03:18:27.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.528+0000 7f7b6effd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 0x7f7b5c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:27.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.528+0000 7f7b6effd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 0x7f7b5c0678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009950 tx=0x7f7b60008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:27.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.532+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7b58070780 con 0x7f7b70101b50 2024-06-06T03:18:27.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.733+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b701083a0 con 0x7f7b70101b50 2024-06-06T03:18:27.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.734+0000 7f7b67fff640 1 -- 172.21.15.153:0/1132472560 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7b58073da0 con 0x7f7b70101b50 2024-06-06T03:18:27.733 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:27.734 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 msgr2=0x7f7b5c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 0x7f7b5c0678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7b60009950 tx=0x7f7b60008040 comp rx=0 tx=0).stop 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 msgr2=0x7f7b70113610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7b58003080 tx=0x7f7b58009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 shutdown_connections 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7b70101b50 0x7f7b70113610 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f7b5c065430 0x7f7b5c0678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 --2- 172.21.15.153:0/1132472560 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f7b701011b0 0x7f7b701130d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 >> 172.21.15.153:0/1132472560 conn(0x7f7b700f6d60 msgr2=0x7f7b700f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 shutdown_connections 2024-06-06T03:18:27.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:27.736+0000 7f7b755bd640 1 -- 172.21.15.153:0/1132472560 wait complete. 2024-06-06T03:18:28.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:27 smithi153 ceph-mon[29300]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:28.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:27 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1132472560' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:27 smithi173 ceph-mon[35650]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:28.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:27 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1132472560' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:30.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:29 smithi153 ceph-mon[29300]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:30.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:29 smithi173 ceph-mon[35650]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:31.351 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:31.618 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:32.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:31 smithi153 ceph-mon[29300]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:32.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:31 smithi173 ceph-mon[35650]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:33.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- 172.21.15.153:0/739660476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac8102c70 msgr2=0x7feac80fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:33.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 --2- 172.21.15.153:0/739660476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac8102c70 0x7feac80fc300 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7feab40048d0 tx=0x7feab4009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- 172.21.15.153:0/739660476 shutdown_connections 2024-06-06T03:18:33.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 --2- 172.21.15.153:0/739660476 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac8102c70 0x7feac80fc300 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 --2- 172.21.15.153:0/739660476 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac81022d0 0x7feac81026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- 172.21.15.153:0/739660476 >> 172.21.15.153:0/739660476 conn(0x7feac80f7e80 msgr2=0x7feac80fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- 172.21.15.153:0/739660476 shutdown_connections 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- 172.21.15.153:0/739660476 wait complete. 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 Processor -- start 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.212+0000 7feacefed640 1 -- start start 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:33.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac8102c70 0x7feac806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac8071520 con 0x7feac81022d0 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac8071690 con 0x7feac8102c70 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57862/0 (socket says 172.21.15.153:57862) 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 -- 172.21.15.153:0/1512767233 learned_addr learned my addr 172.21.15.153:0/1512767233 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feac7fff640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac8102c70 0x7feac806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:33.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 -- 172.21.15.153:0/1512767233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac8102c70 msgr2=0x7feac806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac8102c70 0x7feac806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 -- 172.21.15.153:0/1512767233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4005040 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feaccd89640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feabc002a50 tx=0x7feabc004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feabc007c30 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feabc00f040 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac80702b0 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feabc00dad0 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.213+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac811dcb0 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.214+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac80fda40 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.215+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7feabc00dc70 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.215+0000 7feac57fa640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 0x7feab80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.215+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feabc081bb0 con 0x7feac81022d0 2024-06-06T03:18:33.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.215+0000 7feac7fff640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 0x7feab80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:33.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.216+0000 7feac7fff640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 0x7feab80678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feab4009f80 tx=0x7feab4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:33.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.220+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7feabc04cdf0 con 0x7feac81022d0 2024-06-06T03:18:33.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.421+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feac804b4b0 con 0x7feac81022d0 2024-06-06T03:18:33.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.421+0000 7feac57fa640 1 -- 172.21.15.153:0/1512767233 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7feabc088080 con 0x7feac81022d0 2024-06-06T03:18:33.421 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:33.421 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 msgr2=0x7feab80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 0x7feab80678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7feab4009f80 tx=0x7feab4035040 comp rx=0 tx=0).stop 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 msgr2=0x7feac806f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feabc002a50 tx=0x7feabc004600 comp rx=0 tx=0).stop 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 shutdown_connections 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7feab8065430 0x7feab80678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7feac8102c70 0x7feac806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 --2- 172.21.15.153:0/1512767233 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feac81022d0 0x7feac806f770 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:33.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 >> 172.21.15.153:0/1512767233 conn(0x7feac80f7e80 msgr2=0x7feac80f8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:33.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 shutdown_connections 2024-06-06T03:18:33.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:33.423+0000 7feacefed640 1 -- 172.21.15.153:0/1512767233 wait complete. 2024-06-06T03:18:34.001 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:34.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:18:34.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1512767233' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:34.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:18:34.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:18:34.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:33 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:18:34.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:18:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1512767233' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:18:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:18:34.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:33 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:18:36.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:35 smithi153 ceph-mon[29300]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:36.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:35 smithi173 ceph-mon[35650]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:37.116 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:37.379 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:38.225 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:38 smithi153 ceph-mon[29300]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:38.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:38 smithi173 ceph-mon[35650]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:39.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 -- 172.21.15.153:0/1002826804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27040faa50 msgr2=0x7f27040faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:39.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 --2- 172.21.15.153:0/1002826804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27040faa50 0x7f27040faec0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0048d0 tx=0x7f26ec009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 -- 172.21.15.153:0/1002826804 shutdown_connections 2024-06-06T03:18:39.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 --2- 172.21.15.153:0/1002826804 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27040faa50 0x7f27040faec0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 --2- 172.21.15.153:0/1002826804 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27041023e0 0x7f27041027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 -- 172.21.15.153:0/1002826804 >> 172.21.15.153:0/1002826804 conn(0x7f2704070ab0 msgr2=0x7f2704070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 -- 172.21.15.153:0/1002826804 shutdown_connections 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.068+0000 7f270902d640 1 -- 172.21.15.153:0/1002826804 wait complete. 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 Processor -- start 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 -- start start 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27040faa50 0x7f27041130e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2704113c20 con 0x7f27041023e0 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f270902d640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f270410d060 con 0x7f27040faa50 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f26fbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:39.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f26fbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57872/0 (socket says 172.21.15.153:57872) 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f26fbfff640 1 -- 172.21.15.153:0/1688313453 learned_addr learned my addr 172.21.15.153:0/1688313453 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f2702d9d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27040faa50 0x7f27041130e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.069+0000 7f26fbfff640 1 -- 172.21.15.153:0/1688313453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27040faa50 msgr2=0x7f27041130e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fbfff640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27040faa50 0x7f27041130e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fbfff640 1 -- 172.21.15.153:0/1688313453 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ec005040 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fbfff640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0043d0 tx=0x7f26ec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ec038070 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f270410d2e0 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f26ec032970 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26ec03da60 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.070+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f270410d7a0 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.071+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2704065550 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.071+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f26ec030020 con 0x7f27041023e0 2024-06-06T03:18:39.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.072+0000 7f26fb7fe640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 0x7f26f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:39.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.072+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f26ec0a26d0 con 0x7f27041023e0 2024-06-06T03:18:39.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.072+0000 7f2702d9d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 0x7f26f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:39.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.072+0000 7f2702d9d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 0x7f26f00678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26f4007fd0 tx=0x7f26f4009e90 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:39.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.076+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f26ec070b80 con 0x7f27041023e0 2024-06-06T03:18:39.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.274+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f27040749d0 con 0x7f27041023e0 2024-06-06T03:18:39.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.274+0000 7f26fb7fe640 1 -- 172.21.15.153:0/1688313453 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f26ec0741a0 con 0x7f27041023e0 2024-06-06T03:18:39.275 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:39.275 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 msgr2=0x7f26f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 0x7f26f00678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f26f4007fd0 tx=0x7f26f4009e90 comp rx=0 tx=0).stop 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 msgr2=0x7f2704113620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0043d0 tx=0x7f26ec009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 shutdown_connections 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27041023e0 0x7f2704113620 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f26f0065430 0x7f26f00678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 --2- 172.21.15.153:0/1688313453 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f27040faa50 0x7f27041130e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:39.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 >> 172.21.15.153:0/1688313453 conn(0x7f2704070ab0 msgr2=0x7f27040f9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:39.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 shutdown_connections 2024-06-06T03:18:39.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:39.276+0000 7f270902d640 1 -- 172.21.15.153:0/1688313453 wait complete. 2024-06-06T03:18:40.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:40 smithi153 ceph-mon[29300]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:40.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:40 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1688313453' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:40.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:40 smithi173 ceph-mon[35650]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:40.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:40 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1688313453' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:41.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:41.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:42.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:42 smithi173 ceph-mon[35650]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:42.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:42 smithi153 ceph-mon[29300]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:43.007 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:43.271 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:44.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:44 smithi173 ceph-mon[35650]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:44.368 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:44 smithi153 ceph-mon[29300]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:44.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 -- 172.21.15.153:0/3958114635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fd2b0 msgr2=0x7f462c0fd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:44.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 --2- 172.21.15.153:0/3958114635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fd2b0 0x7f462c0fd680 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4624004900 tx=0x7f4624009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 -- 172.21.15.153:0/3958114635 shutdown_connections 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 --2- 172.21.15.153:0/3958114635 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fdc50 0x7f462c106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 --2- 172.21.15.153:0/3958114635 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fd2b0 0x7f462c0fd680 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 -- 172.21.15.153:0/3958114635 >> 172.21.15.153:0/3958114635 conn(0x7f462c0f6d70 msgr2=0x7f462c0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 -- 172.21.15.153:0/3958114635 shutdown_connections 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.855+0000 7f4633178640 1 -- 172.21.15.153:0/3958114635 wait complete. 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 Processor -- start 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 -- start start 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 0x7f462c1130f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c113c30 con 0x7f462c0fdc50 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f4633178640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c10d070 con 0x7f462c0fd2b0 2024-06-06T03:18:44.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f462bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f462bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57582/0 (socket says 172.21.15.153:57582) 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.856+0000 7f462bfff640 1 -- 172.21.15.153:0/3763234881 learned_addr learned my addr 172.21.15.153:0/3763234881 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f4630f14640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 0x7f462c1130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f462bfff640 1 -- 172.21.15.153:0/3763234881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 msgr2=0x7f462c1130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f462bfff640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 0x7f462c1130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f462bfff640 1 -- 172.21.15.153:0/3763234881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4624005040 con 0x7f462c0fdc50 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f4630f14640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 0x7f462c1130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f462bfff640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f461c00cb10 tx=0x7f461c007ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461c004200 con 0x7f462c0fdc50 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462c10d350 con 0x7f462c0fdc50 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f461c00acf0 con 0x7f462c0fdc50 2024-06-06T03:18:44.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f461c003dc0 con 0x7f462c0fdc50 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.857+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c10d870 con 0x7f462c0fdc50 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.858+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c066030 con 0x7f462c0fdc50 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.858+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f461c00a820 con 0x7f462c0fdc50 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.859+0000 7f46297fa640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 0x7f46180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.859+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f461c082b80 con 0x7f462c0fdc50 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.859+0000 7f4630f14640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 0x7f46180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.859+0000 7f4630f14640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 0x7f46180678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f462402af90 tx=0x7f4624009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:44.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:44.863+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f461c0510f0 con 0x7f462c0fdc50 2024-06-06T03:18:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.066+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f462c10e760 con 0x7f462c0fdc50 2024-06-06T03:18:45.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.066+0000 7f46297fa640 1 -- 172.21.15.153:0/3763234881 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f461c054710 con 0x7f462c0fdc50 2024-06-06T03:18:45.067 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:45.067 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 msgr2=0x7f46180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:45.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 0x7f46180678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f462402af90 tx=0x7f4624009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 msgr2=0x7f462c113630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f461c00cb10 tx=0x7f461c007ee0 comp rx=0 tx=0).stop 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 shutdown_connections 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f462c0fdc50 0x7f462c113630 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f4618065430 0x7f46180678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 --2- 172.21.15.153:0/3763234881 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f462c0fd2b0 0x7f462c1130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 >> 172.21.15.153:0/3763234881 conn(0x7f462c0f6d70 msgr2=0x7f462c0f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 shutdown_connections 2024-06-06T03:18:45.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:45.068+0000 7f4633178640 1 -- 172.21.15.153:0/3763234881 wait complete. 2024-06-06T03:18:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:46 smithi173 ceph-mon[35650]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:46.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:46 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3763234881' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:46.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:46 smithi153 ceph-mon[29300]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:46.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:46 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3763234881' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:48.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:48 smithi173 ceph-mon[35650]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:48.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:48 smithi153 ceph-mon[29300]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:48.675 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:48.942 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:50.334 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:50 smithi153 ceph-mon[29300]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:50.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:50 smithi173 ceph-mon[35650]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 -- 172.21.15.153:0/2807225182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 msgr2=0x7f60300fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 --2- 172.21.15.153:0/2807225182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300fb940 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f60200048d0 tx=0x7f6020009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 -- 172.21.15.153:0/2807225182 shutdown_connections 2024-06-06T03:18:50.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 --2- 172.21.15.153:0/2807225182 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 0x7f60300fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 --2- 172.21.15.153:0/2807225182 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300fb940 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 -- 172.21.15.153:0/2807225182 >> 172.21.15.153:0/2807225182 conn(0x7f6030072110 msgr2=0x7f6030072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 -- 172.21.15.153:0/2807225182 shutdown_connections 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.553+0000 7f6035d89640 1 -- 172.21.15.153:0/2807225182 wait complete. 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 Processor -- start 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 -- start start 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 0x7f60300f8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60300f91b0 con 0x7f60300fb570 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6035d89640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60300f9320 con 0x7f60300fbe80 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54564/0 (socket says 172.21.15.153:54564) 2024-06-06T03:18:50.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 -- 172.21.15.153:0/111361910 learned_addr learned my addr 172.21.15.153:0/111361910 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f6027fff640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 0x7f60300f8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 -- 172.21.15.153:0/111361910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 msgr2=0x7f60300f8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 0x7f60300f8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.554+0000 7f602f7fe640 1 -- 172.21.15.153:0/111361910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6020005040 con 0x7f60300fb570 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f602f7fe640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6020003080 tx=0x7f6020009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6020038070 con 0x7f60300fb570 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6020008ee0 con 0x7f60300fb570 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f602003d870 con 0x7f60300fb570 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60301193d0 con 0x7f60300fb570 2024-06-06T03:18:50.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.555+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6030119700 con 0x7f60300fb570 2024-06-06T03:18:50.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.556+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60300fc920 con 0x7f60300fb570 2024-06-06T03:18:50.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.556+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6020030020 con 0x7f60300fb570 2024-06-06T03:18:50.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.556+0000 7f602cff9640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 0x7f601c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:50.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.557+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f60200a2250 con 0x7f60300fb570 2024-06-06T03:18:50.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.557+0000 7f6027fff640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 0x7f601c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:50.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.557+0000 7f6027fff640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 0x7f601c0678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f60300f9fe0 tx=0x7f6018008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:50.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.561+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6020070780 con 0x7f60300fb570 2024-06-06T03:18:50.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.762+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6030072c20 con 0x7f60300fb570 2024-06-06T03:18:50.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.762+0000 7f602cff9640 1 -- 172.21.15.153:0/111361910 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6020073da0 con 0x7f60300fb570 2024-06-06T03:18:50.763 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:50.763 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:50.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 msgr2=0x7f601c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 0x7f601c0678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f60300f9fe0 tx=0x7f6018008040 comp rx=0 tx=0).stop 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 msgr2=0x7f60300f86a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6020003080 tx=0x7f6020009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 shutdown_connections 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f601c065430 0x7f601c0678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f60300fbe80 0x7f60300f8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 --2- 172.21.15.153:0/111361910 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f60300fb570 0x7f60300f86a0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 >> 172.21.15.153:0/111361910 conn(0x7f6030072110 msgr2=0x7f60300ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 shutdown_connections 2024-06-06T03:18:50.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:50.764+0000 7f6035d89640 1 -- 172.21.15.153:0/111361910 wait complete. 2024-06-06T03:18:51.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:51 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/111361910' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:51.354 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:51 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/111361910' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:52.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:52 smithi173 ceph-mon[35650]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:52.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:52 smithi153 ceph-mon[29300]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:54.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:54 smithi173 ceph-mon[35650]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:54.379 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:18:54.407 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:54 smithi153 ceph-mon[29300]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:54.642 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:18:56.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/1193357110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c48102c70 msgr2=0x7f9c480fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/1193357110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c48102c70 0x7f9c480fc300 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f9c3c0048d0 tx=0x7f9c3c009da0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/1193357110 shutdown_connections 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/1193357110 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c48102c70 0x7f9c480fc300 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/1193357110 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c481022d0 0x7f9c481026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/1193357110 >> 172.21.15.153:0/1193357110 conn(0x7f9c480f7e80 msgr2=0x7f9c480fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/1193357110 shutdown_connections 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/1193357110 wait complete. 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 Processor -- start 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.225+0000 7f9c4e3e2640 1 -- start start 2024-06-06T03:18:56.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c4e3e2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c4e3e2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c48102c70 0x7f9c48115770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c4e3e2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48115d70 con 0x7f9c481022d0 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c4e3e2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4810f1b0 con 0x7f9c48102c70 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54570/0 (socket says 172.21.15.153:54570) 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 -- 172.21.15.153:0/3806231205 learned_addr learned my addr 172.21.15.153:0/3806231205 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c477fe640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c48102c70 0x7f9c48115770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 -- 172.21.15.153:0/3806231205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c48102c70 msgr2=0x7f9c48115770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c48102c70 0x7f9c48115770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 -- 172.21.15.153:0/3806231205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c3c005040 con 0x7f9c481022d0 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c47fff640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9c38002a80 tx=0x7f9c38004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:56.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c38007c30 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9c3800f040 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9c380096b0 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.226+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c4810f490 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.227+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c4810f9b0 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.227+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c480fda40 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.228+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9c3800d6b0 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.228+0000 7f9c44ff9640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 0x7f9c300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.228+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9c38081e10 con 0x7f9c481022d0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.228+0000 7f9c477fe640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 0x7f9c300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.229+0000 7f9c477fe640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 0x7f9c300678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9c48110af0 tx=0x7f9c3c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:18:56.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.232+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9c3804cdf0 con 0x7f9c481022d0 2024-06-06T03:18:56.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:56 smithi173 ceph-mon[35650]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:56.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:56.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.427+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9c480fbac0 con 0x7f9c481022d0 2024-06-06T03:18:56.427 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:56 smithi153 ceph-mon[29300]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:56.427 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:18:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.427+0000 7f9c44ff9640 1 -- 172.21.15.153:0/3806231205 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9c38050340 con 0x7f9c481022d0 2024-06-06T03:18:56.427 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:18:56.428 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 msgr2=0x7f9c300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 0x7f9c300678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9c48110af0 tx=0x7f9c3c035040 comp rx=0 tx=0).stop 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 msgr2=0x7f9c48115230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9c38002a80 tx=0x7f9c38004600 comp rx=0 tx=0).stop 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 shutdown_connections 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9c30065430 0x7f9c300678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9c48102c70 0x7f9c48115770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 --2- 172.21.15.153:0/3806231205 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9c481022d0 0x7f9c48115230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:18:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.429+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 >> 172.21.15.153:0/3806231205 conn(0x7f9c480f7e80 msgr2=0x7f9c48071140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:18:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.430+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 shutdown_connections 2024-06-06T03:18:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:18:56.430+0000 7f9c4e3e2640 1 -- 172.21.15.153:0/3806231205 wait complete. 2024-06-06T03:18:57.060 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:57 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3806231205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:57.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:57 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3806231205' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:18:58.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:18:58 smithi173 ceph-mon[35650]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:18:58.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:18:58 smithi153 ceph-mon[29300]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:00.084 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:00.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:00 smithi173 ceph-mon[35650]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:00.348 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:00.471 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:00 smithi153 ceph-mon[29300]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 -- 172.21.15.153:0/2533323174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a00faa50 msgr2=0x7f59a00faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 --2- 172.21.15.153:0/2533323174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a00faa50 0x7f59a00faec0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f59900048d0 tx=0x7f5990009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 -- 172.21.15.153:0/2533323174 shutdown_connections 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 --2- 172.21.15.153:0/2533323174 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a00faa50 0x7f59a00faec0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 --2- 172.21.15.153:0/2533323174 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a01023e0 0x7f59a01027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:01.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 -- 172.21.15.153:0/2533323174 >> 172.21.15.153:0/2533323174 conn(0x7f59a0070ab0 msgr2=0x7f59a0070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 -- 172.21.15.153:0/2533323174 shutdown_connections 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.963+0000 7f59a616a640 1 -- 172.21.15.153:0/2533323174 wait complete. 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 Processor -- start 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 -- start start 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a00faa50 0x7f59a0113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0113d50 con 0x7f59a01023e0 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f59a616a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a010d100 con 0x7f59a00faa50 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f599effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f599effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37514/0 (socket says 172.21.15.153:37514) 2024-06-06T03:19:01.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f599effd640 1 -- 172.21.15.153:0/3298405423 learned_addr learned my addr 172.21.15.153:0/3298405423 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.964+0000 7f599f7fe640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a00faa50 0x7f59a0113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f599effd640 1 -- 172.21.15.153:0/3298405423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a00faa50 msgr2=0x7f59a0113180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f599effd640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a00faa50 0x7f59a0113180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f599effd640 1 -- 172.21.15.153:0/3298405423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5990005040 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f599effd640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5990003080 tx=0x7f5990009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5990038070 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a010d380 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5990008ee0 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f599003d870 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.965+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a010d840 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.966+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a0065550 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.966+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5990030020 con 0x7f59a01023e0 2024-06-06T03:19:01.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.967+0000 7f5987fff640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 0x7f59880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:01.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.967+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f59900a21b0 con 0x7f59a01023e0 2024-06-06T03:19:01.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.967+0000 7f599f7fe640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 0x7f59880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:01.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.967+0000 7f599f7fe640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 0x7f59880678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5994009950 tx=0x7f5994008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:01.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:01.971+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f59900706e0 con 0x7f59a01023e0 2024-06-06T03:19:02.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.171+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f59a00749d0 con 0x7f59a01023e0 2024-06-06T03:19:02.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.171+0000 7f5987fff640 1 -- 172.21.15.153:0/3298405423 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5990073d00 con 0x7f59a01023e0 2024-06-06T03:19:02.171 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:02.171 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 msgr2=0x7f59880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 0x7f59880678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5994009950 tx=0x7f5994008040 comp rx=0 tx=0).stop 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 msgr2=0x7f59a01136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f5990003080 tx=0x7f5990009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 shutdown_connections 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.173+0000 7f59a616a640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f59a01023e0 0x7f59a01136c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.174+0000 7f59a616a640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5988065430 0x7f59880678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:02.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.174+0000 7f59a616a640 1 --2- 172.21.15.153:0/3298405423 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f59a00faa50 0x7f59a0113180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:02.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.174+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 >> 172.21.15.153:0/3298405423 conn(0x7f59a0070ab0 msgr2=0x7f59a00f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:02.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.174+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 shutdown_connections 2024-06-06T03:19:02.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:02.174+0000 7f59a616a640 1 -- 172.21.15.153:0/3298405423 wait complete. 2024-06-06T03:19:02.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:02 smithi173 ceph-mon[35650]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:02.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:02 smithi153 ceph-mon[29300]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:03.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:03 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3298405423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:03.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:03 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3298405423' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:04.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:04 smithi173 ceph-mon[35650]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:04.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:04 smithi153 ceph-mon[29300]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:05.823 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:06.089 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:06.210 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:06 smithi153 ceph-mon[29300]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:06.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:06 smithi173 ceph-mon[35650]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.726+0000 7f3a12614640 1 -- 172.21.15.153:0/205869646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 msgr2=0x7f3a0c0fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.726+0000 7f3a12614640 1 --2- 172.21.15.153:0/205869646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c0fe2e0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f3a000048d0 tx=0x7f3a00009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 -- 172.21.15.153:0/205869646 shutdown_connections 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 --2- 172.21.15.153:0/205869646 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c0fe2e0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 --2- 172.21.15.153:0/205869646 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 -- 172.21.15.153:0/205869646 >> 172.21.15.153:0/205869646 conn(0x7f3a0c072110 msgr2=0x7f3a0c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 -- 172.21.15.153:0/205869646 shutdown_connections 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 -- 172.21.15.153:0/205869646 wait complete. 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 Processor -- start 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.727+0000 7f3a12614640 1 -- start start 2024-06-06T03:19:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c113b30 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0c10cf70 con 0x7f3a0c0650a0 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:37530/0 (socket says 172.21.15.153:37530) 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 -- 172.21.15.153:0/3045090639 learned_addr learned my addr 172.21.15.153:0/3045090639 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0bfff640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 -- 172.21.15.153:0/3045090639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 msgr2=0x7f3a0c112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 -- 172.21.15.153:0/3045090639 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00005040 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0bfff640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a0b7fe640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3a00003080 tx=0x7f3a00009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a00038070 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3a00008ee0 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a0c10d1f0 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3a0003d870 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.728+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a0c10d730 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.729+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a0c0fda50 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.730+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3a00030020 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.730+0000 7f3a08ff9640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 0x7f39f40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.730+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3a000a2220 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.730+0000 7f3a0bfff640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 0x7f39f40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:07.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.730+0000 7f3a0bfff640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 0x7f39f40678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:07.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.734+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3a00070750 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.930+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3a0c072c20 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.931+0000 7f3a08ff9640 1 -- 172.21.15.153:0/3045090639 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3a00073d70 con 0x7f3a0c0fdeb0 2024-06-06T03:19:07.930 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:07.930 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 msgr2=0x7f39f40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 0x7f39f40678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f39fc007fd0 tx=0x7f39fc009040 comp rx=0 tx=0).stop 2024-06-06T03:19:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 msgr2=0x7f3a0c113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f3a00003080 tx=0x7f3a00009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 shutdown_connections 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3a0c0fdeb0 0x7f3a0c113530 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f39f4065430 0x7f39f40678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 --2- 172.21.15.153:0/3045090639 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f3a0c0650a0 0x7f3a0c112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 >> 172.21.15.153:0/3045090639 conn(0x7f3a0c072110 msgr2=0x7f3a0c100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 shutdown_connections 2024-06-06T03:19:07.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:07.933+0000 7f3a12614640 1 -- 172.21.15.153:0/3045090639 wait complete. 2024-06-06T03:19:08.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:08 smithi153 ceph-mon[29300]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:08.252 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:08 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3045090639' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:08.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:08 smithi173 ceph-mon[35650]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:08.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:08 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3045090639' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:10.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:10 smithi173 ceph-mon[35650]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:10.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:10 smithi153 ceph-mon[29300]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:11.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:11.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:11.553 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:11.820 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:12.231 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:12 smithi153 ceph-mon[29300]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:12.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:12 smithi173 ceph-mon[35650]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 -- 172.21.15.153:0/3713433282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 msgr2=0x7fd9180fe2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/3713433282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd9180fe2e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fd9080048d0 tx=0x7fd908009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 -- 172.21.15.153:0/3713433282 shutdown_connections 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/3713433282 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd9180fe2e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/3713433282 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918065470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 -- 172.21.15.153:0/3713433282 >> 172.21.15.153:0/3713433282 conn(0x7fd918072110 msgr2=0x7fd918072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 -- 172.21.15.153:0/3713433282 shutdown_connections 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 -- 172.21.15.153:0/3713433282 wait complete. 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.507+0000 7fd91f7a0640 1 Processor -- start 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91f7a0640 1 -- start start 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91f7a0640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91f7a0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91f7a0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd918113b30 con 0x7fd9180fdeb0 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91f7a0640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd91810cf70 con 0x7fd9180650a0 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55934/0 (socket says 172.21.15.153:55934) 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 -- 172.21.15.153:0/4260949096 learned_addr learned my addr 172.21.15.153:0/4260949096 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:13.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91d53c640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 -- 172.21.15.153:0/4260949096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 msgr2=0x7fd918112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 -- 172.21.15.153:0/4260949096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd908005040 con 0x7fd9180fdeb0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91d53c640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd91cd3b640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd908003080 tx=0x7fd908009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.508+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd908038070 con 0x7fd9180fdeb0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.509+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd91810d1f0 con 0x7fd9180fdeb0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.509+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd908008ee0 con 0x7fd9180fdeb0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.509+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd90803d870 con 0x7fd9180fdeb0 2024-06-06T03:19:13.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.509+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd91810d730 con 0x7fd9180fdeb0 2024-06-06T03:19:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.510+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9180fda50 con 0x7fd9180fdeb0 2024-06-06T03:19:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.510+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd908030020 con 0x7fd9180fdeb0 2024-06-06T03:19:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.511+0000 7fd90dffb640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 0x7fd9040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.511+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd9080a2250 con 0x7fd9180fdeb0 2024-06-06T03:19:13.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.511+0000 7fd91d53c640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 0x7fd9040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:13.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.511+0000 7fd91d53c640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 0x7fd9040678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd914004be0 tx=0x7fd914009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:13.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.515+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd908070780 con 0x7fd9180fdeb0 2024-06-06T03:19:13.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.713+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd91810e660 con 0x7fd9180fdeb0 2024-06-06T03:19:13.713 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.714+0000 7fd90dffb640 1 -- 172.21.15.153:0/4260949096 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd908073da0 con 0x7fd9180fdeb0 2024-06-06T03:19:13.713 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:13.714 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:13.715 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 msgr2=0x7fd9040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 0x7fd9040678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd914004be0 tx=0x7fd914009040 comp rx=0 tx=0).stop 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 msgr2=0x7fd918113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fd908003080 tx=0x7fd908009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 shutdown_connections 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9180fdeb0 0x7fd918113530 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fd904065430 0x7fd9040678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 --2- 172.21.15.153:0/4260949096 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fd9180650a0 0x7fd918112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 >> 172.21.15.153:0/4260949096 conn(0x7fd918072110 msgr2=0x7fd918100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 shutdown_connections 2024-06-06T03:19:13.716 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:13.716+0000 7fd91f7a0640 1 -- 172.21.15.153:0/4260949096 wait complete. 2024-06-06T03:19:14.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:14 smithi153 ceph-mon[29300]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:14.299 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:14 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4260949096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:14.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:14 smithi173 ceph-mon[35650]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:14.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:14 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4260949096' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:16.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:16 smithi173 ceph-mon[35650]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:16.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:16 smithi153 ceph-mon[29300]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:17.324 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:18.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:18 smithi173 ceph-mon[35650]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:18.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:18 smithi153 ceph-mon[29300]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:19.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- 172.21.15.153:0/1771611277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101310 msgr2=0x7f8a9c1016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 --2- 172.21.15.153:0/1771611277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101310 0x7f8a9c1016e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8a980048d0 tx=0x7f8a98009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- 172.21.15.153:0/1771611277 shutdown_connections 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 --2- 172.21.15.153:0/1771611277 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c0fb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 --2- 172.21.15.153:0/1771611277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101310 0x7f8a9c1016e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- 172.21.15.153:0/1771611277 >> 172.21.15.153:0/1771611277 conn(0x7f8a9c0f6d90 msgr2=0x7f8a9c0f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- 172.21.15.153:0/1771611277 shutdown_connections 2024-06-06T03:19:19.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- 172.21.15.153:0/1771611277 wait complete. 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 Processor -- start 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.187+0000 7f8aa3478640 1 -- start start 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa3478640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 0x7f8a9c113110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa3478640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa3478640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c113c50 con 0x7f8a9c101cb0 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa3478640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9c10d090 con 0x7f8a9c101310 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55952/0 (socket says 172.21.15.153:55952) 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 -- 172.21.15.153:0/393987559 learned_addr learned my addr 172.21.15.153:0/393987559 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 -- 172.21.15.153:0/393987559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 msgr2=0x7f8a9c113110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa1214640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 0x7f8a9c113110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 0x7f8a9c113110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 -- 172.21.15.153:0/393987559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a98005040 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa1214640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 0x7f8a9c113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa0a13640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c002a50 tx=0x7f8a8c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a8c007c30 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8a8c00f040 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9c10d370 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.188+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8a8c00db50 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.189+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9c10d910 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.190+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a9c0fa9f0 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.190+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8a8c01c400 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.190+0000 7f8a91ffb640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.190+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8a8c081e00 con 0x7f8a9c101cb0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.190+0000 7f8aa1214640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.191+0000 7f8aa1214640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 0x7f8a880678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009f80 tx=0x7f8a98009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:19.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.194+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8a8c0502b0 con 0x7f8a9c101cb0 2024-06-06T03:19:19.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.402+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8a9c04b480 con 0x7f8a9c101cb0 2024-06-06T03:19:19.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.402+0000 7f8a91ffb640 1 -- 172.21.15.153:0/393987559 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8a8c0538d0 con 0x7f8a9c101cb0 2024-06-06T03:19:19.402 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:19.402 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 msgr2=0x7f8a880678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 0x7f8a880678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a98009f80 tx=0x7f8a98009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 msgr2=0x7f8a9c113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c002a50 tx=0x7f8a8c004600 comp rx=0 tx=0).stop 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 shutdown_connections 2024-06-06T03:19:19.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a9c101cb0 0x7f8a9c113650 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8a88065430 0x7f8a880678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 --2- 172.21.15.153:0/393987559 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8a9c101310 0x7f8a9c113110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:19.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 >> 172.21.15.153:0/393987559 conn(0x7f8a9c0f6d90 msgr2=0x7f8a9c0f8440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:19.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 shutdown_connections 2024-06-06T03:19:19.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:19.404+0000 7f8aa3478640 1 -- 172.21.15.153:0/393987559 wait complete. 2024-06-06T03:19:20.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:20 smithi173 ceph-mon[35650]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:20.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:20 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/393987559' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:20.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:20 smithi153 ceph-mon[29300]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:20.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:20 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/393987559' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:22.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:22 smithi173 ceph-mon[35650]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:22.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:22 smithi153 ceph-mon[29300]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:23.046 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:24.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:24 smithi173 ceph-mon[35650]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:24.419 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:24 smithi153 ceph-mon[29300]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/2355725812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb01022a0 msgr2=0x7f5bb0102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/2355725812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb01022a0 0x7f5bb0102670 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5ba00048d0 tx=0x7f5ba0009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/2355725812 shutdown_connections 2024-06-06T03:19:24.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/2355725812 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb0102c40 0x7f5bb00fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/2355725812 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb01022a0 0x7f5bb0102670 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/2355725812 >> 172.21.15.153:0/2355725812 conn(0x7f5bb00f7e70 msgr2=0x7f5bb00fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/2355725812 shutdown_connections 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/2355725812 wait complete. 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.907+0000 7f5bb6a1c640 1 Processor -- start 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5bb6a1c640 1 -- start start 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5bb6a1c640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 0x7f5bb0113090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5bb6a1c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5bb6a1c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb0113c60 con 0x7f5bb0102c40 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5bb6a1c640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb010d010 con 0x7f5bb01022a0 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:24.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52368/0 (socket says 172.21.15.153:52368) 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 -- 172.21.15.153:0/1638907323 learned_addr learned my addr 172.21.15.153:0/1638907323 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baffff640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 0x7f5bb0113090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 -- 172.21.15.153:0/1638907323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 msgr2=0x7f5bb0113090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 0x7f5bb0113090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 -- 172.21.15.153:0/1638907323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0005040 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baffff640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 0x7f5bb0113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.908+0000 7f5baf7fe640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400b790 tx=0x7f5ba40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.909+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba400c700 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.909+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ba400cd30 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.909+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb010d2f0 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.909+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ba4013370 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.909+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb010d810 con 0x7f5bb0102c40 2024-06-06T03:19:24.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.910+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb00fda10 con 0x7f5bb0102c40 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.910+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5ba4012070 con 0x7f5bb0102c40 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.910+0000 7f5bacff9640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 0x7f5b980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.911+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5ba4081fe0 con 0x7f5bb0102c40 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.911+0000 7f5baffff640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 0x7f5b980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.911+0000 7f5baffff640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 0x7f5b980678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009f80 tx=0x7f5ba0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:24.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:24.915+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5ba4050510 con 0x7f5bb0102c40 2024-06-06T03:19:25.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.114+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5bb0109490 con 0x7f5bb0102c40 2024-06-06T03:19:25.114 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.115+0000 7f5bacff9640 1 -- 172.21.15.153:0/1638907323 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f5ba404ee60 con 0x7f5bb0102c40 2024-06-06T03:19:25.115 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:25.115 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:25.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 msgr2=0x7f5b980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:25.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 0x7f5b980678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5ba0009f80 tx=0x7f5ba0035040 comp rx=0 tx=0).stop 2024-06-06T03:19:25.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 msgr2=0x7f5bb01135d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:25.116 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400b790 tx=0x7f5ba40091d0 comp rx=0 tx=0).stop 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 shutdown_connections 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5bb0102c40 0x7f5bb01135d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f5b98065430 0x7f5b980678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 --2- 172.21.15.153:0/1638907323 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f5bb01022a0 0x7f5bb0113090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 >> 172.21.15.153:0/1638907323 conn(0x7f5bb00f7e70 msgr2=0x7f5bb00f8d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 shutdown_connections 2024-06-06T03:19:25.117 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:25.117+0000 7f5bb6a1c640 1 -- 172.21.15.153:0/1638907323 wait complete. 2024-06-06T03:19:26.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:26 smithi173 ceph-mon[35650]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:26.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:26 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1638907323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:26.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:26.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:26 smithi153 ceph-mon[29300]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:26.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:26 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1638907323' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:26.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:28.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:28 smithi173 ceph-mon[35650]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:28.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:28 smithi153 ceph-mon[29300]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:28.769 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:29.033 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:30.404 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:30 smithi153 ceph-mon[29300]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:30.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:30 smithi173 ceph-mon[35650]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 -- 172.21.15.153:0/3830659033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 msgr2=0x7f9ed81015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/3830659033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed81015a0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f9ed00048d0 tx=0x7f9ed0009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 -- 172.21.15.153:0/3830659033 shutdown_connections 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/3830659033 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed80fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/3830659033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed81015a0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 -- 172.21.15.153:0/3830659033 >> 172.21.15.153:0/3830659033 conn(0x7f9ed80f6d80 msgr2=0x7f9ed80f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 -- 172.21.15.153:0/3830659033 shutdown_connections 2024-06-06T03:19:30.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.763+0000 7f9edf0c2640 1 -- 172.21.15.153:0/3830659033 wait complete. 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 Processor -- start 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 -- start start 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed810e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed810ec00 con 0x7f9ed81011d0 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edf0c2640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed810ed70 con 0x7f9ed8101b70 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edce5e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edce5e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40416/0 (socket says 172.21.15.153:40416) 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.764+0000 7f9edce5e640 1 -- 172.21.15.153:0/1727514978 learned_addr learned my addr 172.21.15.153:0/1727514978 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9ed7fff640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed810e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edce5e640 1 -- 172.21.15.153:0/1727514978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 msgr2=0x7f9ed810e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edce5e640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed810e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edce5e640 1 -- 172.21.15.153:0/1727514978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed0005040 con 0x7f9ed81011d0 2024-06-06T03:19:30.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9ed7fff640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:19:30.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edce5e640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0003080 tx=0x7f9ed0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ed0038070 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ed0008ee0 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed8117050 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ed003d870 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.765+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed81174c0 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.766+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed80fc940 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.767+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9ed0030020 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.767+0000 7f9ed57fa640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 0x7f9ec40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.767+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9ed00a2240 con 0x7f9ed81011d0 2024-06-06T03:19:30.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.767+0000 7f9ed7fff640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 0x7f9ec40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:30.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.767+0000 7f9ed7fff640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 0x7f9ec40678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80661d0 tx=0x7f9ec8009610 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.771+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9ed0070770 con 0x7f9ed81011d0 2024-06-06T03:19:30.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.968+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9ed804b4e0 con 0x7f9ed81011d0 2024-06-06T03:19:30.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.968+0000 7f9ed57fa640 1 -- 172.21.15.153:0/1727514978 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9ed0073d90 con 0x7f9ed81011d0 2024-06-06T03:19:30.968 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:30.968 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 msgr2=0x7f9ec40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 0x7f9ec40678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9ed80661d0 tx=0x7f9ec8009610 comp rx=0 tx=0).stop 2024-06-06T03:19:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 msgr2=0x7f9ed810dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f9ed0003080 tx=0x7f9ed0009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 shutdown_connections 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ec4065430 0x7f9ec40678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ed8101b70 0x7f9ed810e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 --2- 172.21.15.153:0/1727514978 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ed81011d0 0x7f9ed810dfe0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 >> 172.21.15.153:0/1727514978 conn(0x7f9ed80f6d80 msgr2=0x7f9ed80f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 shutdown_connections 2024-06-06T03:19:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:30.971+0000 7f9edf0c2640 1 -- 172.21.15.153:0/1727514978 wait complete. 2024-06-06T03:19:31.251 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:31 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1727514978' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:31.388 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:31 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1727514978' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:32.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:32 smithi153 ceph-mon[29300]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:32.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:32 smithi173 ceph-mon[35650]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:34.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:34 smithi153 ceph-mon[29300]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:34.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:34 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:19:34.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:34 smithi173 ceph-mon[35650]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:34.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:34 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-06T03:19:34.753 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:35.009 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:35.409 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:35 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:19:35.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:35 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:19:35.410 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:35 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:19:35.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:35 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-06T03:19:35.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:35 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-06T03:19:35.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:35 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' 2024-06-06T03:19:36.422 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:36 smithi153 ceph-mon[29300]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:36.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:36 smithi173 ceph-mon[35650]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:36.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- 172.21.15.153:0/3576146177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 msgr2=0x7f3094104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:36.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 --2- 172.21.15.153:0/3576146177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f3094104c20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f30900048d0 tx=0x7f3090009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- 172.21.15.153:0/3576146177 shutdown_connections 2024-06-06T03:19:36.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 --2- 172.21.15.153:0/3576146177 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 0x7f309406e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 --2- 172.21.15.153:0/3576146177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f3094104c20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- 172.21.15.153:0/3576146177 >> 172.21.15.153:0/3576146177 conn(0x7f3094068810 msgr2=0x7f3094068c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- 172.21.15.153:0/3576146177 shutdown_connections 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- 172.21.15.153:0/3576146177 wait complete. 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 Processor -- start 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.745+0000 7f309c3fd640 1 -- start start 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f309c3fd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 0x7f309411b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f309c3fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f309c3fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f309411c360 con 0x7f3094104850 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f309c3fd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3094115770 con 0x7f309406e570 2024-06-06T03:19:36.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40420/0 (socket says 172.21.15.153:40420) 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 -- 172.21.15.153:0/427212183 learned_addr learned my addr 172.21.15.153:0/427212183 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f309a199640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 0x7f309411b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 -- 172.21.15.153:0/427212183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 msgr2=0x7f309411b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 0x7f309411b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 -- 172.21.15.153:0/427212183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3090005040 con 0x7f3094104850 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f3099998640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f308400c830 tx=0x7f3084004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f308400a7d0 con 0x7f3094104850 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.746+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f308400ae00 con 0x7f3094104850 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.747+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3094115a50 con 0x7f3094104850 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.747+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f30840113a0 con 0x7f3094104850 2024-06-06T03:19:36.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.747+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3094115ff0 con 0x7f3094104850 2024-06-06T03:19:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.748+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f309406e110 con 0x7f3094104850 2024-06-06T03:19:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.748+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3084012400 con 0x7f3094104850 2024-06-06T03:19:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.748+0000 7f308affd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 0x7f30800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:36.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.748+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f30840824e0 con 0x7f3094104850 2024-06-06T03:19:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.748+0000 7f309a199640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 0x7f30800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:36.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.749+0000 7f309a199640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 0x7f30800678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3090009f80 tx=0x7f3090035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:36.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.753+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3084050ab0 con 0x7f3094104850 2024-06-06T03:19:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.952+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3094111c60 con 0x7f3094104850 2024-06-06T03:19:36.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.953+0000 7f308affd640 1 -- 172.21.15.153:0/427212183 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f30840540d0 con 0x7f3094104850 2024-06-06T03:19:36.952 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:36.952 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 msgr2=0x7f30800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 0x7f30800678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3090009f80 tx=0x7f3090035040 comp rx=0 tx=0).stop 2024-06-06T03:19:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 msgr2=0x7f309411bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:36.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f308400c830 tx=0x7f3084004430 comp rx=0 tx=0).stop 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 shutdown_connections 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3094104850 0x7f309411bd10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f3080065430 0x7f30800678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 --2- 172.21.15.153:0/427212183 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f309406e570 0x7f309411b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 >> 172.21.15.153:0/427212183 conn(0x7f3094068810 msgr2=0x7f309410ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 shutdown_connections 2024-06-06T03:19:36.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:36.955+0000 7f309c3fd640 1 -- 172.21.15.153:0/427212183 wait complete. 2024-06-06T03:19:37.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:37 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/427212183' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:37.712 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:37 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/427212183' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:38.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:38 smithi173 ceph-mon[35650]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:38.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:38 smithi153 ceph-mon[29300]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:40.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:40 smithi173 ceph-mon[35650]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:40.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:40.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:40 smithi153 ceph-mon[29300]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:41.005 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:41.398 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:41 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:41.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:41 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:42.451 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:42 smithi153 ceph-mon[29300]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:42.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:42 smithi173 ceph-mon[35650]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:42.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.790+0000 7fbe29b17640 1 -- 172.21.15.153:0/3331655185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 msgr2=0x7fbe241015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.790+0000 7fbe29b17640 1 --2- 172.21.15.153:0/3331655185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe241015a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fbe140048d0 tx=0x7fbe14009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 -- 172.21.15.153:0/3331655185 shutdown_connections 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 --2- 172.21.15.153:0/3331655185 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe240fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 --2- 172.21.15.153:0/3331655185 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe241015a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 -- 172.21.15.153:0/3331655185 >> 172.21.15.153:0/3331655185 conn(0x7fbe240f6d80 msgr2=0x7fbe240f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 -- 172.21.15.153:0/3331655185 shutdown_connections 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 -- 172.21.15.153:0/3331655185 wait complete. 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 Processor -- start 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 -- start start 2024-06-06T03:19:42.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.791+0000 7fbe29b17640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe241135b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe29b17640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe24113bb0 con 0x7fbe241011d0 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe29b17640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2410cff0 con 0x7fbe24101b70 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48154/0 (socket says 172.21.15.153:48154) 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 -- 172.21.15.153:0/4280938624 learned_addr learned my addr 172.21.15.153:0/4280938624 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe22ffd640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe241135b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 -- 172.21.15.153:0/4280938624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 msgr2=0x7fbe241135b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe241135b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 -- 172.21.15.153:0/4280938624 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe14005040 con 0x7fbe241011d0 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe22ffd640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe241135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe237fe640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbe14003080 tx=0x7fbe14009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:42.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe14038070 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2410d270 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbe14008ee0 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe1403d870 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.792+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2410d730 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.793+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe240fc940 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.794+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbe14030020 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.794+0000 7fbe0bfff640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 0x7fbe0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.794+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbe140a2240 con 0x7fbe241011d0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.794+0000 7fbe22ffd640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 0x7fbe0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:42.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.795+0000 7fbe22ffd640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 0x7fbe0c0678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbe18007fd0 tx=0x7fbe18009040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:42.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.798+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbe14070770 con 0x7fbe241011d0 2024-06-06T03:19:42.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.997+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbe240fa9c0 con 0x7fbe241011d0 2024-06-06T03:19:42.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.997+0000 7fbe0bfff640 1 -- 172.21.15.153:0/4280938624 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbe14073d90 con 0x7fbe241011d0 2024-06-06T03:19:42.998 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:42.998 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.999+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 msgr2=0x7fbe0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.999+0000 7fbe29b17640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 0x7fbe0c0678e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fbe18007fd0 tx=0x7fbe18009040 comp rx=0 tx=0).stop 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.999+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 msgr2=0x7fbe24113070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:42.999+0000 7fbe29b17640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fbe14003080 tx=0x7fbe14009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 shutdown_connections 2024-06-06T03:19:43.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fbe0c065430 0x7fbe0c0678e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:43.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fbe24101b70 0x7fbe241135b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:43.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 --2- 172.21.15.153:0/4280938624 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbe241011d0 0x7fbe24113070 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:43.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 >> 172.21.15.153:0/4280938624 conn(0x7fbe240f6d80 msgr2=0x7fbe240f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:43.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 shutdown_connections 2024-06-06T03:19:43.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:43.000+0000 7fbe29b17640 1 -- 172.21.15.153:0/4280938624 wait complete. 2024-06-06T03:19:43.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:43 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/4280938624' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:43.676 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:43 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/4280938624' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:44.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:44 smithi173 ceph-mon[35650]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:44.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:44 smithi153 ceph-mon[29300]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:46.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:46 smithi173 ceph-mon[35650]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:46.613 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:46 smithi153 ceph-mon[29300]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:46.701 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:46.967 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:48.470 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:48 smithi153 ceph-mon[29300]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:48.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:48 smithi173 ceph-mon[35650]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 -- 172.21.15.153:0/980430482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fdc30 msgr2=0x7f9ae0106300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/980430482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fdc30 0x7f9ae0106300 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4004900 tx=0x7f9ad4009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 -- 172.21.15.153:0/980430482 shutdown_connections 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/980430482 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fdc30 0x7f9ae0106300 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/980430482 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.681+0000 7f9ae77f4640 1 -- 172.21.15.153:0/980430482 >> 172.21.15.153:0/980430482 conn(0x7f9ae00f6d90 msgr2=0x7f9ae00f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:48.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 -- 172.21.15.153:0/980430482 shutdown_connections 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 -- 172.21.15.153:0/980430482 wait complete. 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 Processor -- start 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 -- start start 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fdc30 0x7f9ae00fb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00fbc60 con 0x7f9ae00fd290 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae77f4640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00fbdd0 con 0x7f9ae00fdc30 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae5590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae5590640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:48160/0 (socket says 172.21.15.153:48160) 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.682+0000 7f9ae5590640 1 -- 172.21.15.153:0/3667493615 learned_addr learned my addr 172.21.15.153:0/3667493615 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae4d8f640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fdc30 0x7f9ae00fb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae5590640 1 -- 172.21.15.153:0/3667493615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fdc30 msgr2=0x7f9ae00fb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:48.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae5590640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fdc30 0x7f9ae00fb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae5590640 1 -- 172.21.15.153:0/3667493615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad4005040 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae5590640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800bbb0 tx=0x7f9ad8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad800c9c0 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9ad80092e0 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae0119430 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9ad8007640 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.683+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae0119870 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.684+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae0065df0 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.684+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9ad8012070 con 0x7f9ae00fd290 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.685+0000 7f9addffb640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 0x7f9ac80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.685+0000 7f9ae4d8f640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 0x7f9ac80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:48.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.685+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9ad8082460 con 0x7f9ae00fd290 2024-06-06T03:19:48.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.686+0000 7f9ae4d8f640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 0x7f9ac80678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40047f0 tx=0x7f9ad4009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:48.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.690+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9ad8050990 con 0x7f9ae00fd290 2024-06-06T03:19:48.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.890+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9ae00fa990 con 0x7f9ae00fd290 2024-06-06T03:19:48.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.890+0000 7f9addffb640 1 -- 172.21.15.153:0/3667493615 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9ad8053fb0 con 0x7f9ae00fd290 2024-06-06T03:19:48.891 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:48.891 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 msgr2=0x7f9ac80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 0x7f9ac80678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40047f0 tx=0x7f9ad4009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 msgr2=0x7f9ae00fb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f9ad800bbb0 tx=0x7f9ad8009840 comp rx=0 tx=0).stop 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 shutdown_connections 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f9ac8065430 0x7f9ac80678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f9ae00fdc30 0x7f9ae00fb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 --2- 172.21.15.153:0/3667493615 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f9ae00fd290 0x7f9ae00fb150 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 >> 172.21.15.153:0/3667493615 conn(0x7f9ae00f6d90 msgr2=0x7f9ae00f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.892+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 shutdown_connections 2024-06-06T03:19:48.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:48.893+0000 7f9ae77f4640 1 -- 172.21.15.153:0/3667493615 wait complete. 2024-06-06T03:19:49.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:49 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3667493615' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:49.588 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:49 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3667493615' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:50.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:50 smithi173 ceph-mon[35650]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:50.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:50 smithi153 ceph-mon[29300]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:52.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:52 smithi173 ceph-mon[35650]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:52.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:52.676 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:52 smithi153 ceph-mon[29300]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:52.911 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:19:54.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:54 smithi173 ceph-mon[35650]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:54.634 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:54 smithi153 ceph-mon[29300]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- 172.21.15.153:0/2697270347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f41022f0 msgr2=0x7fb8f41026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 --2- 172.21.15.153:0/2697270347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f41022f0 0x7fb8f41026c0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40048d0 tx=0x7fb8e4009da0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- 172.21.15.153:0/2697270347 shutdown_connections 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 --2- 172.21.15.153:0/2697270347 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f4102c90 0x7fb8f40fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 --2- 172.21.15.153:0/2697270347 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f41022f0 0x7fb8f41026c0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- 172.21.15.153:0/2697270347 >> 172.21.15.153:0/2697270347 conn(0x7fb8f4070a40 msgr2=0x7fb8f4070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- 172.21.15.153:0/2697270347 shutdown_connections 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- 172.21.15.153:0/2697270347 wait complete. 2024-06-06T03:19:54.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 Processor -- start 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.634+0000 7fb8fa878640 1 -- start start 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8fa878640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f41022f0 0x7fb8f41130a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8fa878640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8fa878640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f4113be0 con 0x7fb8f4102c90 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8fa878640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f410d020 con 0x7fb8f41022f0 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56420/0 (socket says 172.21.15.153:56420) 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 -- 172.21.15.153:0/3274440569 learned_addr learned my addr 172.21.15.153:0/3274440569 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f3fff640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f41022f0 0x7fb8f41130a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 -- 172.21.15.153:0/3274440569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f41022f0 msgr2=0x7fb8f41130a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f41022f0 0x7fb8f41130a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 -- 172.21.15.153:0/3274440569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4005040 con 0x7fb8f4102c90 2024-06-06T03:19:54.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f37fe640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c760 tx=0x7fb8e800cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e800a820 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8e800ae50 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f410d300 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.635+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8e80115e0 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.636+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f410d820 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.636+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f4065510 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.637+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb8e8012400 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.637+0000 7fb8f0ff9640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 0x7fb8dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.637+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb8e80826c0 con 0x7fb8f4102c90 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.638+0000 7fb8f3fff640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 0x7fb8dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.638+0000 7fb8f3fff640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 0x7fb8dc0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009f80 tx=0x7fb8e4035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:19:54.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.641+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb8e8050cb0 con 0x7fb8f4102c90 2024-06-06T03:19:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.840+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8f4074960 con 0x7fb8f4102c90 2024-06-06T03:19:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.840+0000 7fb8f0ff9640 1 -- 172.21.15.153:0/3274440569 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb8e80126b0 con 0x7fb8f4102c90 2024-06-06T03:19:54.841 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:19:54.841 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:19:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 msgr2=0x7fb8dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 0x7fb8dc0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb8e4009f80 tx=0x7fb8e4035040 comp rx=0 tx=0).stop 2024-06-06T03:19:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 msgr2=0x7fb8f41135e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:19:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb8e800c760 tx=0x7fb8e800cb60 comp rx=0 tx=0).stop 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 shutdown_connections 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb8f4102c90 0x7fb8f41135e0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb8dc065430 0x7fb8dc0678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 --2- 172.21.15.153:0/3274440569 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb8f41022f0 0x7fb8f41130a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 >> 172.21.15.153:0/3274440569 conn(0x7fb8f4070a40 msgr2=0x7fb8f41050b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 shutdown_connections 2024-06-06T03:19:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:19:54.842+0000 7fb8fa878640 1 -- 172.21.15.153:0/3274440569 wait complete. 2024-06-06T03:19:55.501 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:55 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3274440569' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:55.587 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:55 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3274440569' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:19:56.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:56 smithi153 ceph-mon[29300]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:56.612 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:56 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:56.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:56 smithi173 ceph-mon[35650]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:56.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:56 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:19:58.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:19:58.662 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:19:58 smithi153 ceph-mon[29300]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:58.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:19:58 smithi173 ceph-mon[35650]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:19:58.898 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:00.624 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:00 smithi153 ceph-mon[29300]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:00.625 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:00 smithi153 ceph-mon[29300]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T03:20:00.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 -- 172.21.15.153:0/2923476984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc1023e0 msgr2=0x7fe5dc1027b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:00.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2923476984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1027b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc0048d0 tx=0x7fe5cc009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 -- 172.21.15.153:0/2923476984 shutdown_connections 2024-06-06T03:20:00.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2923476984 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc0faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2923476984 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1027b0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 -- 172.21.15.153:0/2923476984 >> 172.21.15.153:0/2923476984 conn(0x7fe5dc070ab0 msgr2=0x7fe5dc070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 -- 172.21.15.153:0/2923476984 shutdown_connections 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 -- 172.21.15.153:0/2923476984 wait complete. 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.625+0000 7fe5e347e640 1 Processor -- start 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 -- start start 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1136a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc113d30 con 0x7fe5dc0faa50 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc10d0e0 con 0x7fe5dc1023e0 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53776/0 (socket says 172.21.15.153:53776) 2024-06-06T03:20:00.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 -- 172.21.15.153:0/2964045891 learned_addr learned my addr 172.21.15.153:0/2964045891 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e0a19640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1136a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 -- 172.21.15.153:0/2964045891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 msgr2=0x7fe5dc1136a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1136a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 -- 172.21.15.153:0/2964045891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5cc005040 con 0x7fe5dc0faa50 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e0a19640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e121a640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc003080 tx=0x7fe5cc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc038070 con 0x7fe5dc0faa50 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe5cc008ee0 con 0x7fe5dc0faa50 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5dc10d360 con 0x7fe5dc0faa50 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.626+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe5cc03d870 con 0x7fe5dc0faa50 2024-06-06T03:20:00.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.627+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc10d820 con 0x7fe5dc0faa50 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.628+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc065550 con 0x7fe5dc0faa50 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.628+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fe5cc030020 con 0x7fe5dc0faa50 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.629+0000 7fe5d1ffb640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 0x7fe5c80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.629+0000 7fe5e0a19640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 0x7fe5c80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.629+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe5cc0a21b0 con 0x7fe5dc0faa50 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.629+0000 7fe5e0a19640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 0x7fe5c80678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc10ea20 tx=0x7fe5d8008040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:00.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.633+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe5cc0706e0 con 0x7fe5dc0faa50 2024-06-06T03:20:00.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.830+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe5dc0749d0 con 0x7fe5dc0faa50 2024-06-06T03:20:00.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.831+0000 7fe5d1ffb640 1 -- 172.21.15.153:0/2964045891 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fe5cc073d00 con 0x7fe5dc0faa50 2024-06-06T03:20:00.830 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:00.830 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:00.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 msgr2=0x7fe5c80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:00.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 0x7fe5c80678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fe5dc10ea20 tx=0x7fe5d8008040 comp rx=0 tx=0).stop 2024-06-06T03:20:00.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 msgr2=0x7fe5dc113160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:00.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fe5cc003080 tx=0x7fe5cc009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 shutdown_connections 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fe5c8065430 0x7fe5c80678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fe5dc1023e0 0x7fe5dc1136a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 --2- 172.21.15.153:0/2964045891 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe5dc0faa50 0x7fe5dc113160 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 >> 172.21.15.153:0/2964045891 conn(0x7fe5dc070ab0 msgr2=0x7fe5dc0f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 shutdown_connections 2024-06-06T03:20:00.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:00.833+0000 7fe5e347e640 1 -- 172.21.15.153:0/2964045891 wait complete. 2024-06-06T03:20:00.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:00 smithi173 ceph-mon[35650]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:00.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:00 smithi173 ceph-mon[35650]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-06T03:20:01.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:01 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/2964045891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:01.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:01 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/2964045891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:02.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:02 smithi153 ceph-mon[29300]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:02.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:02 smithi173 ceph-mon[35650]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:04.635 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:20:04.664 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:04 smithi153 ceph-mon[29300]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:04.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:04 smithi173 ceph-mon[35650]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:04.901 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:06.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:06 smithi153 ceph-mon[29300]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 -- 172.21.15.153:0/1710630165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6481011c0 msgr2=0x7fb648101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 --2- 172.21.15.153:0/1710630165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6481011c0 0x7fb648101590 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb6380048d0 tx=0x7fb638009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 -- 172.21.15.153:0/1710630165 shutdown_connections 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 --2- 172.21.15.153:0/1710630165 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb648101b60 0x7fb6480fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 --2- 172.21.15.153:0/1710630165 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6481011c0 0x7fb648101590 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 -- 172.21.15.153:0/1710630165 >> 172.21.15.153:0/1710630165 conn(0x7fb6480f6d70 msgr2=0x7fb6480f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 -- 172.21.15.153:0/1710630165 shutdown_connections 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.755+0000 7fb64e593640 1 -- 172.21.15.153:0/1710630165 wait complete. 2024-06-06T03:20:06.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 Processor -- start 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 -- start start 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 0x7fb648112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb648113b30 con 0x7fb648101b60 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb64e593640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64810cf70 con 0x7fb6481011c0 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb6477fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:53792/0 (socket says 172.21.15.153:53792) 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.756+0000 7fb6477fe640 1 -- 172.21.15.153:0/913356221 learned_addr learned my addr 172.21.15.153:0/913356221 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb647fff640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 0x7fb648112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb6477fe640 1 -- 172.21.15.153:0/913356221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 msgr2=0x7fb648112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb6477fe640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 0x7fb648112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb6477fe640 1 -- 172.21.15.153:0/913356221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb638005040 con 0x7fb648101b60 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb647fff640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 0x7fb648112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb6477fe640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb630002a50 tx=0x7fb630004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb630007c30 con 0x7fb648101b60 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb63000f040 con 0x7fb648101b60 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64810d250 con 0x7fb648101b60 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb63000db50 con 0x7fb648101b60 2024-06-06T03:20:06.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.757+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64810d7f0 con 0x7fb648101b60 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.758+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6480fc930 con 0x7fb648101b60 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.758+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb63001c400 con 0x7fb648101b60 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.759+0000 7fb644ff9640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 0x7fb6340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.759+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb630081e40 con 0x7fb648101b60 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.759+0000 7fb647fff640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 0x7fb6340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:06.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.759+0000 7fb647fff640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 0x7fb6340678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb638009f80 tx=0x7fb638035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:06.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.763+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb63004cdf0 con 0x7fb648101b60 2024-06-06T03:20:06.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:06 smithi173 ceph-mon[35650]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.964+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb648108320 con 0x7fb648101b60 2024-06-06T03:20:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.965+0000 7fb644ff9640 1 -- 172.21.15.153:0/913356221 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb6300502f0 con 0x7fb648101b60 2024-06-06T03:20:06.964 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:06.964 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 msgr2=0x7fb6340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 0x7fb6340678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb638009f80 tx=0x7fb638035040 comp rx=0 tx=0).stop 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 msgr2=0x7fb648113530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb630002a50 tx=0x7fb630004600 comp rx=0 tx=0).stop 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 shutdown_connections 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb648101b60 0x7fb648113530 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fb634065430 0x7fb6340678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 --2- 172.21.15.153:0/913356221 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fb6481011c0 0x7fb648112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:06.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 >> 172.21.15.153:0/913356221 conn(0x7fb6480f6d70 msgr2=0x7fb6480f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:06.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 shutdown_connections 2024-06-06T03:20:06.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:06.967+0000 7fb64e593640 1 -- 172.21.15.153:0/913356221 wait complete. 2024-06-06T03:20:07.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:07 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/913356221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:07.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:07 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/913356221' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:08.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:08 smithi153 ceph-mon[29300]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:08.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:08 smithi173 ceph-mon[35650]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:10.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:10 smithi153 ceph-mon[29300]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:10.818 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:20:10.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:10 smithi173 ceph-mon[35650]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:11.074 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:11.469 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:11 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:20:11.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:11 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:20:12.507 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:12 smithi153 ceph-mon[29300]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:12.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.820+0000 7f81f53bd640 1 -- 172.21.15.153:0/4285138412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 msgr2=0x7f81f0101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:12.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.820+0000 7f81f53bd640 1 --2- 172.21.15.153:0/4285138412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f0101620 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f81e00048d0 tx=0x7f81e0009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:12.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 -- 172.21.15.153:0/4285138412 shutdown_connections 2024-06-06T03:20:12.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 --2- 172.21.15.153:0/4285138412 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 0x7f81f00fbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:12.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 --2- 172.21.15.153:0/4285138412 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f0101620 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:12.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 -- 172.21.15.153:0/4285138412 >> 172.21.15.153:0/4285138412 conn(0x7f81f0070a40 msgr2=0x7f81f0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:12.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 -- 172.21.15.153:0/4285138412 shutdown_connections 2024-06-06T03:20:12.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 -- 172.21.15.153:0/4285138412 wait complete. 2024-06-06T03:20:12.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 Processor -- start 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.821+0000 7f81f53bd640 1 -- start start 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81f53bd640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 0x7f81f010dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81f53bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81f53bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f010eb30 con 0x7f81f0101250 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81f53bd640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f010eca0 con 0x7f81f00fbaf0 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59022/0 (socket says 172.21.15.153:59022) 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 -- 172.21.15.153:0/3016635697 learned_addr learned my addr 172.21.15.153:0/3016635697 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81eeffd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 0x7f81f010dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 -- 172.21.15.153:0/3016635697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 msgr2=0x7f81f010dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:12.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 0x7f81f010dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 -- 172.21.15.153:0/3016635697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0005040 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81ee7fc640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c790 tx=0x7f81e40043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e400a7a0 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f81e400add0 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f0117010 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.822+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f81e40115e0 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.823+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f0117520 con 0x7f81f0101250 2024-06-06T03:20:12.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.824+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81f00fcb00 con 0x7f81f0101250 2024-06-06T03:20:12.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.824+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f81e4012400 con 0x7f81f0101250 2024-06-06T03:20:12.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.824+0000 7f81d77fe640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 0x7f81d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:12.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.824+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f81e40824c0 con 0x7f81f0101250 2024-06-06T03:20:12.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.824+0000 7f81eeffd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 0x7f81d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:12.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.825+0000 7f81eeffd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 0x7f81d80678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009f80 tx=0x7f81e0035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:12.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:12.829+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f81e4050b50 con 0x7f81f0101250 2024-06-06T03:20:12.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:12 smithi173 ceph-mon[35650]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.030+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f81f01083d0 con 0x7f81f0101250 2024-06-06T03:20:13.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.030+0000 7f81d77fe640 1 -- 172.21.15.153:0/3016635697 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f81e4054170 con 0x7f81f0101250 2024-06-06T03:20:13.030 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:13.030 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 msgr2=0x7f81d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 0x7f81d80678e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009f80 tx=0x7f81e0035040 comp rx=0 tx=0).stop 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 msgr2=0x7f81f010e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f81e400c790 tx=0x7f81e40043c0 comp rx=0 tx=0).stop 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 shutdown_connections 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f81f0101250 0x7f81f010e4e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f81d8065430 0x7f81d80678e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 --2- 172.21.15.153:0/3016635697 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f81f00fbaf0 0x7f81f010dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 >> 172.21.15.153:0/3016635697 conn(0x7f81f0070a40 msgr2=0x7f81f00f8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 shutdown_connections 2024-06-06T03:20:13.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:13.032+0000 7f81f53bd640 1 -- 172.21.15.153:0/3016635697 wait complete. 2024-06-06T03:20:13.741 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:13 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3016635697' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:13.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:13 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3016635697' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:14.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:14 smithi153 ceph-mon[29300]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:14.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:14 smithi173 ceph-mon[35650]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:16.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:16 smithi153 ceph-mon[29300]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:16.764 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:20:16.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:16 smithi173 ceph-mon[35650]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:17.029 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:18.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:18 smithi153 ceph-mon[29300]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:18.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:18 smithi173 ceph-mon[35650]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 -- 172.21.15.153:0/1465488548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fd310 msgr2=0x7f22300fd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 --2- 172.21.15.153:0/1465488548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fd310 0x7f22300fd6e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f222c0048d0 tx=0x7f222c009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 -- 172.21.15.153:0/1465488548 shutdown_connections 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 --2- 172.21.15.153:0/1465488548 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fdcb0 0x7f2230106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 --2- 172.21.15.153:0/1465488548 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fd310 0x7f22300fd6e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 -- 172.21.15.153:0/1465488548 >> 172.21.15.153:0/1465488548 conn(0x7f22300f6db0 msgr2=0x7f22300f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:18.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 -- 172.21.15.153:0/1465488548 shutdown_connections 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.872+0000 7f2237ae6640 1 -- 172.21.15.153:0/1465488548 wait complete. 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 Processor -- start 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 -- start start 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fd310 0x7f2230113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230113bc0 con 0x7f22300fdcb0 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2237ae6640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223010d000 con 0x7f22300fd310 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59052/0 (socket says 172.21.15.153:59052) 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 -- 172.21.15.153:0/371542325 learned_addr learned my addr 172.21.15.153:0/371542325 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235882640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fd310 0x7f2230113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 -- 172.21.15.153:0/371542325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fd310 msgr2=0x7f2230113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:18.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fd310 0x7f2230113080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 -- 172.21.15.153:0/371542325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c005040 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f2235081640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f222000c810 tx=0x7f22200043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.873+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f222000a7a0 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.874+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f222000add0 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.874+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223010d2e0 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.874+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f22200115c0 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.874+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223010d800 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.874+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2230065ff0 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.875+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2220012420 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.875+0000 7f22267fc640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 0x7f221c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.875+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2220082970 con 0x7f22300fdcb0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.875+0000 7f2235882640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 0x7f221c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:18.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.876+0000 7f2235882640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 0x7f221c0678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f222c000c00 tx=0x7f222c035040 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:18.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:18.879+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2220050e20 con 0x7f22300fdcb0 2024-06-06T03:20:19.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.075+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f223010e6f0 con 0x7f22300fdcb0 2024-06-06T03:20:19.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.075+0000 7f22267fc640 1 -- 172.21.15.153:0/371542325 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2220054440 con 0x7f22300fdcb0 2024-06-06T03:20:19.075 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:19.075 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:19.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 msgr2=0x7f221c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:19.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 0x7f221c0678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f222c000c00 tx=0x7f222c035040 comp rx=0 tx=0).stop 2024-06-06T03:20:19.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 msgr2=0x7f22301135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:19.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f222000c810 tx=0x7f22200043c0 comp rx=0 tx=0).stop 2024-06-06T03:20:19.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 shutdown_connections 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f22300fdcb0 0x7f22301135c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f221c065430 0x7f221c0678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 --2- 172.21.15.153:0/371542325 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f22300fd310 0x7f2230113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 >> 172.21.15.153:0/371542325 conn(0x7f22300f6db0 msgr2=0x7f2230100c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 shutdown_connections 2024-06-06T03:20:19.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:19.078+0000 7f2237ae6640 1 -- 172.21.15.153:0/371542325 wait complete. 2024-06-06T03:20:19.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:19 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/371542325' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:19.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:19 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/371542325' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:20.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:20 smithi153 ceph-mon[29300]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:20.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:20 smithi173 ceph-mon[35650]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:22.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:22 smithi153 ceph-mon[29300]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:22.825 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:20:22.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:22 smithi173 ceph-mon[35650]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:23.092 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:24.502 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:24 smithi153 ceph-mon[29300]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 -- 172.21.15.153:0/388443064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8514101b50 msgr2=0x7f85140fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 --2- 172.21.15.153:0/388443064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8514101b50 0x7f85140fb1e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f84fc0048d0 tx=0x7f84fc009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 -- 172.21.15.153:0/388443064 shutdown_connections 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 --2- 172.21.15.153:0/388443064 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8514101b50 0x7f85140fb1e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 --2- 172.21.15.153:0/388443064 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f85141011b0 0x7f8514101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 -- 172.21.15.153:0/388443064 >> 172.21.15.153:0/388443064 conn(0x7f85140f6d60 msgr2=0x7f85140f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:24.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 -- 172.21.15.153:0/388443064 shutdown_connections 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.732+0000 7f8518862640 1 -- 172.21.15.153:0/388443064 wait complete. 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 Processor -- start 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 -- start start 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8514101b50 0x7f85140fd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140fd920 con 0x7f85141011b0 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f8518862640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85140fda90 con 0x7f8514101b50 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:51774/0 (socket says 172.21.15.153:51774) 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 -- 172.21.15.153:0/1656839622 learned_addr learned my addr 172.21.15.153:0/1656839622 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 -- 172.21.15.153:0/1656839622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8514101b50 msgr2=0x7f85140fd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8514101b50 0x7f85140fd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.733+0000 7f851259c640 1 -- 172.21.15.153:0/1656839622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc005040 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f851259c640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f850400c790 tx=0x7f85040043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f850400a820 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f850400ae50 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8514119390 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f85040115e0 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.734+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85141197d0 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.735+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85140fc920 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.735+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8504012400 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.736+0000 7f850affd640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 0x7f85000678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.736+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8504082690 con 0x7f85141011b0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.736+0000 7f850bfff640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 0x7f85000678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:24.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.736+0000 7f850bfff640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 0x7f85000678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84fc032430 tx=0x7f84fc009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:24.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.740+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8504050bc0 con 0x7f85141011b0 2024-06-06T03:20:24.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:24 smithi173 ceph-mon[35650]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.940+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f85140fa9a0 con 0x7f85141011b0 2024-06-06T03:20:24.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.940+0000 7f850affd640 1 -- 172.21.15.153:0/1656839622 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f85040541e0 con 0x7f85141011b0 2024-06-06T03:20:24.939 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:24.940 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:24.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 msgr2=0x7f85000678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 0x7f85000678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f84fc032430 tx=0x7f84fc009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 msgr2=0x7f85140fce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f850400c790 tx=0x7f85040043c0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 shutdown_connections 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7f8500065430 0x7f85000678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7f8514101b50 0x7f85140fd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 --2- 172.21.15.153:0/1656839622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85141011b0 0x7f85140fce50 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 >> 172.21.15.153:0/1656839622 conn(0x7f85140f6d60 msgr2=0x7f85140657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 shutdown_connections 2024-06-06T03:20:24.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:24.943+0000 7f8518862640 1 -- 172.21.15.153:0/1656839622 wait complete. 2024-06-06T03:20:25.649 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:25 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/1656839622' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:25.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:25 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/1656839622' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:26.751 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:26 smithi153 ceph-mon[29300]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:26.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:26 smithi153 ceph-mon[29300]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:20:26.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:26 smithi173 ceph-mon[35650]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:26.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:26 smithi173 ceph-mon[35650]: from='mgr.14212 172.21.15.153:0/3107497525' entity='mgr.smithi153.gidtue' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-06T03:20:28.697 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-06-06T03:20:28.727 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:28 smithi153 ceph-mon[29300]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:28.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:28 smithi173 ceph-mon[35650]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:28.965 INFO:teuthology.orchestra.run.smithi153.stderr:Inferring config /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/mon.smithi153/config 2024-06-06T03:20:29.752 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:29 smithi153 ceph-mon[29300]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:29.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:29 smithi173 ceph-mon[35650]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:30.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 -- 172.21.15.153:0/3793745574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101b50 msgr2=0x7fce700fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:30.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 --2- 172.21.15.153:0/3793745574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101b50 0x7fce700fb1e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fce580048d0 tx=0x7fce58009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 -- 172.21.15.153:0/3793745574 shutdown_connections 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 --2- 172.21.15.153:0/3793745574 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce70101b50 0x7fce700fb1e0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 --2- 172.21.15.153:0/3793745574 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce701011b0 0x7fce70101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 -- 172.21.15.153:0/3793745574 >> 172.21.15.153:0/3793745574 conn(0x7fce700f6d60 msgr2=0x7fce700f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 -- 172.21.15.153:0/3793745574 shutdown_connections 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.762+0000 7fce7538a640 1 -- 172.21.15.153:0/3793745574 wait complete. 2024-06-06T03:20:30.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 Processor -- start 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 -- start start 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 --2- >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 0x7fce7010e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7010eb90 con 0x7fce701011b0 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce7538a640 1 -- --> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7010ed00 con 0x7fce70101b50 2024-06-06T03:20:30.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:45618/0 (socket says 172.21.15.153:45618) 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 -- 172.21.15.153:0/3079859723 learned_addr learned my addr 172.21.15.153:0/3079859723 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6e7fc640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 0x7fce7010e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 -- 172.21.15.153:0/3079859723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 msgr2=0x7fce7010e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 0x7fce7010e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6effd640 1 -- 172.21.15.153:0/3079859723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce58005040 con 0x7fce701011b0 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.763+0000 7fce6e7fc640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 0x7fce7010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce6effd640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fce6800db00 tx=0x7fce68004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce68017070 con 0x7fce701011b0 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce70117070 con 0x7fce701011b0 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fce6800f040 con 0x7fce701011b0 2024-06-06T03:20:30.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fce68003ae0 con 0x7fce701011b0 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.764+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce70117580 con 0x7fce701011b0 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.765+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce700fc920 con 0x7fce701011b0 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.765+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fce68003c40 con 0x7fce701011b0 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.766+0000 7fce677fe640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 0x7fce5c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.766+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fce680825c0 con 0x7fce701011b0 2024-06-06T03:20:30.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.766+0000 7fce6e7fc640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 0x7fce5c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-06T03:20:30.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.766+0000 7fce6e7fc640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 0x7fce5c0678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fce58009f80 tx=0x7fce58009da0 comp rx=0 tx=0).ready entity=mgr.14212 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-06T03:20:30.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.770+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fce68050a70 con 0x7fce701011b0 2024-06-06T03:20:30.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.967+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce700fa9a0 con 0x7fce701011b0 2024-06-06T03:20:30.968 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-06-06T03:20:30.968 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.968+0000 7fce677fe640 1 -- 172.21.15.153:0/3079859723 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fce68054090 con 0x7fce701011b0 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 msgr2=0x7fce5c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 0x7fce5c0678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fce58009f80 tx=0x7fce58009da0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 msgr2=0x7fce7010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fce6800db00 tx=0x7fce68004e60 comp rx=0 tx=0).stop 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 shutdown_connections 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:6800/2373762556,v1:172.21.15.153:6801/2373762556] conn(0x7fce5c065430 0x7fce5c0678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.173:3300/0,v1:172.21.15.173:6789/0] conn(0x7fce70101b50 0x7fce7010e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 --2- 172.21.15.153:0/3079859723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fce701011b0 0x7fce7010e000 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-06T03:20:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 >> 172.21.15.153:0/3079859723 conn(0x7fce700f6d60 msgr2=0x7fce700f7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-06T03:20:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 shutdown_connections 2024-06-06T03:20:30.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-06-06T03:20:30.970+0000 7fce7538a640 1 -- 172.21.15.153:0/3079859723 wait complete. 2024-06-06T03:20:31.745 INFO:tasks.cephadm:Teardown begin 2024-06-06T03:20:31.746 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T03:20:31.748 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:20:31.813 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:20:31.848 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-06T03:20:31.848 DEBUG:teuthology.orchestra.run.smithi153:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T03:20:31.870 DEBUG:teuthology.orchestra.run.smithi173:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-06T03:20:31.906 INFO:tasks.cephadm:Stopping all daemons... 2024-06-06T03:20:31.906 INFO:tasks.cephadm.mon.smithi153:Stopping mon.smithi153... 2024-06-06T03:20:31.906 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153 2024-06-06T03:20:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:31 smithi153 ceph-mon[29300]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:31 smithi153 ceph-mon[29300]: from='client.? 172.21.15.153:0/3079859723' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:32.002 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:31 smithi153 systemd[1]: Stopping Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:32.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:31 smithi173 ceph-mon[35650]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-06T03:20:32.338 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:31 smithi173 ceph-mon[35650]: from='client.? 172.21.15.153:0/3079859723' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-06T03:20:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153[29296]: 2024-06-06T03:20:32.294+0000 7fcadb3d9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi153 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153[29296]: 2024-06-06T03:20:32.294+0000 7fcadb3d9640 -1 mon.smithi153@0(leader) e2 *** Got Signal Terminated *** 2024-06-06T03:20:32.626 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 podman[80277]: 2024-06-06 03:20:32.355190825 +0000 UTC m=+0.256153377 container died 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892) 2024-06-06T03:20:32.910 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 podman[80277]: 2024-06-06 03:20:32.726573822 +0000 UTC m=+0.627536365 container cleanup 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-06T03:20:32.910 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 bash[80277]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153 2024-06-06T03:20:33.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service' 2024-06-06T03:20:33.182 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:32 smithi153 podman[80330]: 2024-06-06 03:20:32.911129161 +0000 UTC m=+0.559386304 container remove 7fe78125aa682ee6c246de09a572c2038e06cf4635b7026c976536b50c103ba3 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi153, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-6ee6892, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-06T03:20:33.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:33 smithi153 systemd[1]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service: Deactivated successfully. 2024-06-06T03:20:33.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:33 smithi153 systemd[1]: Stopped Ceph mon.smithi153 for 0b80117e-23b2-11ef-bc9b-c7b262605968. 2024-06-06T03:20:33.183 INFO:journalctl@ceph.mon.smithi153.smithi153.stdout:Jun 06 03:20:33 smithi153 systemd[1]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi153.service: Consumed 8.118s CPU time. 2024-06-06T03:20:33.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:20:33.207 INFO:tasks.cephadm.mon.smithi153:Stopped mon.smithi153 2024-06-06T03:20:33.207 INFO:tasks.cephadm.mon.smithi173:Stopping mon.smithi173... 2024-06-06T03:20:33.207 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi173 2024-06-06T03:20:33.528 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:33 smithi173 systemd[1]: Stopping Ceph mon.smithi173 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:33.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:33 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173[35607]: 2024-06-06T03:20:33.528+0000 7efd79600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi173 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:33.837 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:33 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173[35607]: 2024-06-06T03:20:33.528+0000 7efd79600640 -1 mon.smithi173@1(peon) e2 *** Got Signal Terminated *** 2024-06-06T03:20:33.838 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:33 smithi173 podman[60033]: 2024-06-06 03:20:33.692860872 +0000 UTC m=+0.275473066 container died 1495eb01b886fee8d62198a967345b6517ce00908c8087d03a09269fac4dcd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-6ee6892, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-06T03:20:34.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:34 smithi173 podman[60033]: 2024-06-06 03:20:34.130667593 +0000 UTC m=+0.713279805 container cleanup 1495eb01b886fee8d62198a967345b6517ce00908c8087d03a09269fac4dcd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-06T03:20:34.337 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:34 smithi173 bash[60033]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173 2024-06-06T03:20:34.611 INFO:journalctl@ceph.mon.smithi173.smithi173.stdout:Jun 06 03:20:34 smithi173 podman[60047]: 2024-06-06 03:20:34.357995123 +0000 UTC m=+0.667009365 container remove 1495eb01b886fee8d62198a967345b6517ce00908c8087d03a09269fac4dcd08 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-mon-smithi173, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2) 2024-06-06T03:20:34.616 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@mon.smithi173.service' 2024-06-06T03:20:34.657 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:20:34.657 INFO:tasks.cephadm.mon.smithi173:Stopped mon.smithi173 2024-06-06T03:20:34.657 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-06T03:20:34.657 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.0 2024-06-06T03:20:34.979 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:34 smithi153 systemd[1]: Stopping Ceph osd.0 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:35.251 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:34 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0[47221]: 2024-06-06T03:20:34.980+0000 7f66cbaa7640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:35.252 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:34 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0[47221]: 2024-06-06T03:20:34.980+0000 7f66cbaa7640 -1 osd.0 40 *** Got signal Terminated *** 2024-06-06T03:20:35.252 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:34 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0[47221]: 2024-06-06T03:20:34.980+0000 7f66cbaa7640 -1 osd.0 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:20:40.606 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:40 smithi153 podman[80497]: 2024-06-06 03:20:40.298579866 +0000 UTC m=+5.442359385 container died b05d374e43b384bebbdd6e30f6c3379ce9ac4e498122b368fdc5ca3941b5a36d (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0) 2024-06-06T03:20:41.001 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:40 smithi153 podman[80497]: 2024-06-06 03:20:40.822532307 +0000 UTC m=+5.966311820 container cleanup b05d374e43b384bebbdd6e30f6c3379ce9ac4e498122b368fdc5ca3941b5a36d (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-06T03:20:41.002 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:40 smithi153 bash[80497]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0 2024-06-06T03:20:41.446 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80526]: 2024-06-06 03:20:41.024061259 +0000 UTC m=+0.714417506 container remove b05d374e43b384bebbdd6e30f6c3379ce9ac4e498122b368fdc5ca3941b5a36d (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-06-06T03:20:41.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.350843923 +0000 UTC m=+0.017366459 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:20:41.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.474919618 +0000 UTC m=+0.141442197 container create 0a29b4154a4723573453ea8126ef568fb9b0c8e6c3680f1e7fe4ce97fcdcc088 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, ceph=True) 2024-06-06T03:20:41.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.67718532 +0000 UTC m=+0.343707844 container init 0a29b4154a4723573453ea8126ef568fb9b0c8e6c3680f1e7fe4ce97fcdcc088 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T03:20:41.752 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.680469893 +0000 UTC m=+0.346992417 container start 0a29b4154a4723573453ea8126ef568fb9b0c8e6c3680f1e7fe4ce97fcdcc088 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, ceph=True, GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T03:20:42.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.78461623 +0000 UTC m=+0.451138768 container attach 0a29b4154a4723573453ea8126ef568fb9b0c8e6c3680f1e7fe4ce97fcdcc088 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.35.3) 2024-06-06T03:20:42.238 INFO:journalctl@ceph.osd.0.smithi153.stdout:Jun 06 03:20:41 smithi153 podman[80614]: 2024-06-06 03:20:41.931277084 +0000 UTC m=+0.597799621 container died 0a29b4154a4723573453ea8126ef568fb9b0c8e6c3680f1e7fe4ce97fcdcc088 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:20:42.639 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.0.service' 2024-06-06T03:20:42.680 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:20:42.680 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-06T03:20:42.680 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-06T03:20:42.680 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.1 2024-06-06T03:20:43.001 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:42 smithi153 systemd[1]: Stopping Ceph osd.1 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:43.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:43 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1[54547]: 2024-06-06T03:20:43.027+0000 7fbf6872e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:43.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:43 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1[54547]: 2024-06-06T03:20:43.027+0000 7fbf6872e640 -1 osd.1 40 *** Got signal Terminated *** 2024-06-06T03:20:43.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:43 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1[54547]: 2024-06-06T03:20:43.027+0000 7fbf6872e640 -1 osd.1 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:20:48.752 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:48 smithi153 podman[80716]: 2024-06-06 03:20:48.443900993 +0000 UTC m=+5.540296887 container died f51e0cef7cc0266c1eccb8aa8f1d90599b8d270f16bf40bde75f9011c3dc7fb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, ceph=True, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-06T03:20:49.175 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:48 smithi153 podman[80716]: 2024-06-06 03:20:48.92531328 +0000 UTC m=+6.021709193 container cleanup f51e0cef7cc0266c1eccb8aa8f1d90599b8d270f16bf40bde75f9011c3dc7fb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3) 2024-06-06T03:20:49.175 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:48 smithi153 bash[80716]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1 2024-06-06T03:20:49.502 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80732]: 2024-06-06 03:20:49.176456233 +0000 UTC m=+0.731637501 container remove f51e0cef7cc0266c1eccb8aa8f1d90599b8d270f16bf40bde75f9011c3dc7fb2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3) 2024-06-06T03:20:49.795 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80822]: 2024-06-06 03:20:49.44773209 +0000 UTC m=+0.016796616 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:20:49.795 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80822]: 2024-06-06 03:20:49.57724873 +0000 UTC m=+0.146313239 container create 0c4427043a0a795a6de3b3334036fece5272d4215fba7d16dcae3fdae15a4cd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1-deactivate, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:20:49.795 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80822]: 2024-06-06 03:20:49.796173702 +0000 UTC m=+0.365238211 container init 0c4427043a0a795a6de3b3334036fece5272d4215fba7d16dcae3fdae15a4cd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, GIT_BRANCH=HEAD) 2024-06-06T03:20:50.057 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80822]: 2024-06-06 03:20:49.799954689 +0000 UTC m=+0.369019198 container start 0c4427043a0a795a6de3b3334036fece5272d4215fba7d16dcae3fdae15a4cd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True) 2024-06-06T03:20:50.057 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:49 smithi153 podman[80822]: 2024-06-06 03:20:49.887774645 +0000 UTC m=+0.456839169 container attach 0c4427043a0a795a6de3b3334036fece5272d4215fba7d16dcae3fdae15a4cd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2) 2024-06-06T03:20:50.390 INFO:journalctl@ceph.osd.1.smithi153.stdout:Jun 06 03:20:50 smithi153 podman[80822]: 2024-06-06 03:20:50.058402798 +0000 UTC m=+0.627467310 container died 0c4427043a0a795a6de3b3334036fece5272d4215fba7d16dcae3fdae15a4cd5 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240528, io.buildah.version=1.35.3) 2024-06-06T03:20:50.807 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.1.service' 2024-06-06T03:20:50.853 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:20:50.853 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-06T03:20:50.854 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-06T03:20:50.854 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.2 2024-06-06T03:20:51.151 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:50 smithi153 systemd[1]: Stopping Ceph osd.2 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:51.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:51 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2[62236]: 2024-06-06T03:20:51.151+0000 7f9ed79cd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:51.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:51 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2[62236]: 2024-06-06T03:20:51.151+0000 7f9ed79cd640 -1 osd.2 40 *** Got signal Terminated *** 2024-06-06T03:20:51.502 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:51 smithi153 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2[62236]: 2024-06-06T03:20:51.151+0000 7f9ed79cd640 -1 osd.2 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:20:56.752 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:56 smithi153 podman[80924]: 2024-06-06 03:20:56.492246022 +0000 UTC m=+5.443746580 container died a651759ad723bcad5657dcc8ddde7024943ca7516a89c6fafe8af80aa152e5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T03:20:57.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:56 smithi153 podman[80924]: 2024-06-06 03:20:56.977053917 +0000 UTC m=+5.928554463 container cleanup a651759ad723bcad5657dcc8ddde7024943ca7516a89c6fafe8af80aa152e5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T03:20:57.227 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:56 smithi153 bash[80924]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2 2024-06-06T03:20:57.501 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[80940]: 2024-06-06 03:20:57.228265517 +0000 UTC m=+0.734753237 container remove a651759ad723bcad5657dcc8ddde7024943ca7516a89c6fafe8af80aa152e5fb (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-06T03:20:57.880 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[81028]: 2024-06-06 03:20:57.500255914 +0000 UTC m=+0.017625889 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:20:57.881 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[81028]: 2024-06-06 03:20:57.612629528 +0000 UTC m=+0.129999475 container create e7f46681a45794f06a96e1c21ceb5f6d0e734f758524df1f9cd62ad2a3f21a62 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True) 2024-06-06T03:20:57.881 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[81028]: 2024-06-06 03:20:57.814871353 +0000 UTC m=+0.332241307 container init e7f46681a45794f06a96e1c21ceb5f6d0e734f758524df1f9cd62ad2a3f21a62 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2-deactivate, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240528, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:20:57.881 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[81028]: 2024-06-06 03:20:57.818224478 +0000 UTC m=+0.335594432 container start e7f46681a45794f06a96e1c21ceb5f6d0e734f758524df1f9cd62ad2a3f21a62 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892) 2024-06-06T03:20:57.881 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:57 smithi153 podman[81028]: 2024-06-06 03:20:57.881434863 +0000 UTC m=+0.398804825 container attach e7f46681a45794f06a96e1c21ceb5f6d0e734f758524df1f9cd62ad2a3f21a62 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-06-06T03:20:58.252 INFO:journalctl@ceph.osd.2.smithi153.stdout:Jun 06 03:20:58 smithi153 podman[81028]: 2024-06-06 03:20:58.068177147 +0000 UTC m=+0.585547095 container died e7f46681a45794f06a96e1c21ceb5f6d0e734f758524df1f9cd62ad2a3f21a62 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240528, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T03:20:58.651 DEBUG:teuthology.orchestra.run.smithi153:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.2.service' 2024-06-06T03:20:58.690 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:20:58.693 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-06T03:20:58.693 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-06T03:20:58.693 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.3 2024-06-06T03:20:59.050 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:20:58 smithi173 systemd[1]: Stopping Ceph osd.3 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:20:59.337 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:20:59 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:20:59.051+0000 7f093ee1d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:20:59.337 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:20:59 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:20:59.051+0000 7f093ee1d640 -1 osd.3 40 *** Got signal Terminated *** 2024-06-06T03:20:59.337 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:20:59 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:20:59.051+0000 7f093ee1d640 -1 osd.3 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:21:01.388 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:01 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:01.088+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:02.587 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:02 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:02.126+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:03.587 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:03 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:03.165+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:03.938 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:03 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:03.655+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:04.337 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:03 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3[41114]: 2024-06-06T03:21:03.939+0000 7f093a433640 -1 osd.3 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:34.388883+0000 front 2024-06-06T03:20:34.388906+0000 (oldest deadline 2024-06-06T03:21:03.189297+0000) 2024-06-06T03:21:04.338 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:04 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:04.120+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:04.750 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:04 smithi173 podman[60155]: 2024-06-06 03:21:04.465594225 +0000 UTC m=+5.576114974 container died 77c901d18bb69f6993b65fe3bbcb2701fa100dcf0f29a1f334e11b5ef39ef9b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528) 2024-06-06T03:21:04.750 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:04 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:04.694+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:05.084 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:04 smithi173 podman[60155]: 2024-06-06 03:21:04.850856507 +0000 UTC m=+5.961377259 container cleanup 77c901d18bb69f6993b65fe3bbcb2701fa100dcf0f29a1f334e11b5ef39ef9b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, org.label-schema.license=GPLv2) 2024-06-06T03:21:05.084 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:04 smithi173 bash[60155]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3 2024-06-06T03:21:05.337 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:05 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:05.105+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:05.338 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60169]: 2024-06-06 03:21:05.085865709 +0000 UTC m=+0.613189058 container remove 77c901d18bb69f6993b65fe3bbcb2701fa100dcf0f29a1f334e11b5ef39ef9b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3, org.label-schema.build-date=20240528, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3) 2024-06-06T03:21:05.698 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.327164184 +0000 UTC m=+0.016991507 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:21:05.698 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.428736162 +0000 UTC m=+0.118563486 container create 33b995a1266184826e68df77202bfa3b683179edbcbcff2db41e3bcf00ec8667 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-06T03:21:05.698 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.648466413 +0000 UTC m=+0.338293722 container init 33b995a1266184826e68df77202bfa3b683179edbcbcff2db41e3bcf00ec8667 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-06-06T03:21:05.698 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.651680798 +0000 UTC m=+0.341508106 container start 33b995a1266184826e68df77202bfa3b683179edbcbcff2db41e3bcf00ec8667 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-06-06T03:21:06.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:05 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:05.698+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:06.088 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.733951747 +0000 UTC m=+0.423779066 container attach 33b995a1266184826e68df77202bfa3b683179edbcbcff2db41e3bcf00ec8667 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS) 2024-06-06T03:21:06.088 INFO:journalctl@ceph.osd.3.smithi173.stdout:Jun 06 03:21:05 smithi173 podman[60257]: 2024-06-06 03:21:05.897489107 +0000 UTC m=+0.587316423 container died 33b995a1266184826e68df77202bfa3b683179edbcbcff2db41e3bcf00ec8667 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.35.3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=quincy-6ee6892) 2024-06-06T03:21:06.451 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:06 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:06.100+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:06.474 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.3.service' 2024-06-06T03:21:06.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:21:06.516 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-06T03:21:06.517 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-06T03:21:06.517 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.4 2024-06-06T03:21:06.730 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:06 smithi173 systemd[1]: Stopping Ceph osd.4 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:21:06.731 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:06 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:06.654+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:07.059 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:06 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:06.864+0000 7f2bb6e3b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:21:07.059 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:06 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:06.864+0000 7f2bb6e3b640 -1 osd.4 40 *** Got signal Terminated *** 2024-06-06T03:21:07.059 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:06 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:06.864+0000 7f2bb6e3b640 -1 osd.4 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:21:07.059 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:07 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:07.059+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:08.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:07 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:07.637+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:08.587 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:08 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:08.089+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:09.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:08 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:08.674+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:09.587 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:09 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:09.132+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:09.588 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:09 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:09.132+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.276737+0000 front 2024-06-06T03:20:47.276724+0000 (oldest deadline 2024-06-06T03:21:08.976562+0000) 2024-06-06T03:21:10.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:09 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:09.704+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:10.587 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:10 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:10.115+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:10.588 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:10 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:10.115+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.276737+0000 front 2024-06-06T03:20:47.276724+0000 (oldest deadline 2024-06-06T03:21:08.976562+0000) 2024-06-06T03:21:11.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:10 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:10.666+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:11.388 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:11 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:11.107+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:36.775702+0000 front 2024-06-06T03:20:36.775677+0000 (oldest deadline 2024-06-06T03:21:00.875564+0000) 2024-06-06T03:21:11.389 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:11 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4[47923]: 2024-06-06T03:21:11.107+0000 7f2bb044d640 -1 osd.4 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.276737+0000 front 2024-06-06T03:20:47.276724+0000 (oldest deadline 2024-06-06T03:21:08.976562+0000) 2024-06-06T03:21:12.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:11 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:11.717+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:12.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:11 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:11.717+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:12.472 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:12 smithi173 podman[60361]: 2024-06-06 03:21:12.20943678 +0000 UTC m=+5.454833046 container died 4bde94658e8ee80d5ecf2ec8ee4eb4a924694d4cd834682ae58b1242a0a86ef2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T03:21:12.837 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:12 smithi173 podman[60361]: 2024-06-06 03:21:12.56498704 +0000 UTC m=+5.810383299 container cleanup 4bde94658e8ee80d5ecf2ec8ee4eb4a924694d4cd834682ae58b1242a0a86ef2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, org.label-schema.build-date=20240528, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True) 2024-06-06T03:21:12.838 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:12 smithi173 bash[60361]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4 2024-06-06T03:21:12.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:12 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:12.672+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:12.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:12 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:12.672+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:13.188 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:12 smithi173 podman[60386]: 2024-06-06 03:21:12.841652511 +0000 UTC m=+0.630198731 container remove 4bde94658e8ee80d5ecf2ec8ee4eb4a924694d4cd834682ae58b1242a0a86ef2 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d) 2024-06-06T03:21:13.469 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.091784946 +0000 UTC m=+0.016647266 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:21:13.469 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.201115245 +0000 UTC m=+0.125977546 container create 8ae255c75c967896504f4b1cd70decc89ef64b6aa4074d99b5e62595e10b9534 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240528) 2024-06-06T03:21:13.469 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.41209186 +0000 UTC m=+0.336954181 container init 8ae255c75c967896504f4b1cd70decc89ef64b6aa4074d99b5e62595e10b9534 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-6ee6892, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-06T03:21:13.470 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.415730435 +0000 UTC m=+0.340592738 container start 8ae255c75c967896504f4b1cd70decc89ef64b6aa4074d99b5e62595e10b9534 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.35.3, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, maintainer=Guillaume Abrioux , RELEASE=quincy-6ee6892, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, CEPH_POINT_RELEASE=) 2024-06-06T03:21:13.470 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.470345522 +0000 UTC m=+0.395207829 container attach 8ae255c75c967896504f4b1cd70decc89ef64b6aa4074d99b5e62595e10b9534 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-06-06T03:21:13.837 INFO:journalctl@ceph.osd.4.smithi173.stdout:Jun 06 03:21:13 smithi173 podman[60479]: 2024-06-06 03:21:13.676247011 +0000 UTC m=+0.601109320 container died 8ae255c75c967896504f4b1cd70decc89ef64b6aa4074d99b5e62595e10b9534 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-4-deactivate, RELEASE=quincy-6ee6892, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240528, GIT_BRANCH=HEAD, io.buildah.version=1.35.3, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:21:13.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:13 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:13.630+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:13.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:13 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:13.630+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:14.240 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.4.service' 2024-06-06T03:21:14.282 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:21:14.282 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-06T03:21:14.282 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-06T03:21:14.283 DEBUG:teuthology.orchestra.run.smithi173:> sudo systemctl stop ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.5 2024-06-06T03:21:14.587 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 systemd[1]: Stopping Ceph osd.5 for 0b80117e-23b2-11ef-bc9b-c7b262605968... 2024-06-06T03:21:15.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:14.617+0000 7ff07e54b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-06T03:21:15.088 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:14.617+0000 7ff07e54b640 -1 osd.5 40 *** Got signal Terminated *** 2024-06-06T03:21:15.088 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:14.617+0000 7ff07e54b640 -1 osd.5 40 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-06T03:21:15.088 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:14.642+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:15.088 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:14 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:14.642+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:16.087 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:15 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:15.609+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:16.088 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:15 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:15.609+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:16.837 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:16 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:16.575+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:16.895 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:16 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:16.575+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:17.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:17 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:17.537+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:17.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:17 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:17.537+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:18.837 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:18 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:18.540+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:18.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:18 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:18.540+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:19.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:19 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:19.512+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6806 osd.0 since back 2024-06-06T03:20:37.479004+0000 front 2024-06-06T03:20:37.478937+0000 (oldest deadline 2024-06-06T03:21:03.378909+0000) 2024-06-06T03:21:19.838 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:19 smithi173 ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5[54767]: 2024-06-06T03:21:19.512+0000 7ff07bb65640 -1 osd.5 40 heartbeat_check: no reply from 172.21.15.153:6814 osd.1 since back 2024-06-06T03:20:47.980198+0000 front 2024-06-06T03:20:47.980249+0000 (oldest deadline 2024-06-06T03:21:10.880085+0000) 2024-06-06T03:21:20.392 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 podman[60581]: 2024-06-06 03:21:20.093590301 +0000 UTC m=+5.585733584 container died 10645602a1ef92bc6d43de14c0374b59f1779c979c8aee7a6bf82a1ed3c95b00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:21:20.392 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 podman[60581]: 2024-06-06 03:21:20.393208076 +0000 UTC m=+5.885351350 container cleanup 10645602a1ef92bc6d43de14c0374b59f1779c979c8aee7a6bf82a1ed3c95b00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=quincy-6ee6892, io.buildah.version=1.35.3, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240528) 2024-06-06T03:21:20.837 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 bash[60581]: ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5 2024-06-06T03:21:20.837 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 podman[60595]: 2024-06-06 03:21:20.603237529 +0000 UTC m=+0.507266960 container remove 10645602a1ef92bc6d43de14c0374b59f1779c979c8aee7a6bf82a1ed3c95b00 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892) 2024-06-06T03:21:21.223 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 podman[60687]: 2024-06-06 03:21:20.8460148 +0000 UTC m=+0.017407023 image pull 7513f4a6eabdd63260b5c11f68ee60613052ca5a66bba63b7ac6a38728d7b79a quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a 2024-06-06T03:21:21.223 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:20 smithi173 podman[60687]: 2024-06-06 03:21:20.962781949 +0000 UTC m=+0.134174179 container create 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-6ee6892, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240528) 2024-06-06T03:21:21.223 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:21 smithi173 podman[60687]: 2024-06-06 03:21:21.148643833 +0000 UTC m=+0.320036051 container init 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-06T03:21:21.223 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:21 smithi173 podman[60687]: 2024-06-06 03:21:21.151714374 +0000 UTC m=+0.323106582 container start 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.35.3, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, RELEASE=quincy-6ee6892, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:21:21.223 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:21 smithi173 podman[60687]: 2024-06-06 03:21:21.224010265 +0000 UTC m=+0.395402474 container attach 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.35.3, RELEASE=quincy-6ee6892, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528) 2024-06-06T03:21:21.587 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:21 smithi173 podman[60687]: 2024-06-06 03:21:21.395265153 +0000 UTC m=+0.566657360 container died 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240528, org.label-schema.schema-version=1.0, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.35.3, org.label-schema.license=GPLv2, RELEASE=quincy-6ee6892, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-06-06T03:21:21.953 INFO:journalctl@ceph.osd.5.smithi173.stdout:Jun 06 03:21:21 smithi173 podman[60711]: 2024-06-06 03:21:21.935967342 +0000 UTC m=+0.532877080 container remove 4489f8453879495ba67c91655d1893ef4d48462764bee8dedbc9f0c6f1bc1424 (image=quay.ceph.io/ceph-ci/ceph@sha256:f72fe1939e59e1f17bc3674a20e6062d72bb718e24c720b12206544f1493a26a, name=ceph-0b80117e-23b2-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-6ee6892, GIT_CLEAN=True, GIT_COMMIT=34dce1d36b472526edf5c42fc1eec9ea2a84070d, ceph=True, io.buildah.version=1.35.3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240528, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-06T03:21:21.959 DEBUG:teuthology.orchestra.run.smithi173:> sudo pkill -f 'journalctl -f -n 0 -u ceph-0b80117e-23b2-11ef-bc9b-c7b262605968@osd.5.service' 2024-06-06T03:21:21.997 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-06T03:21:21.998 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-06T03:21:21.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T03:21:31.007 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 --force --keep-logs 2024-06-06T03:21:35.989 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:21:36.022 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-06T03:21:36.050 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-06T03:21:36.051 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi153/crash 2024-06-06T03:21:36.052 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T03:21:36.103 INFO:teuthology.orchestra.run.smithi153.stderr:tar: /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T03:21:36.104 INFO:teuthology.orchestra.run.smithi153.stderr:tar: Error is not recoverable: exiting now 2024-06-06T03:21:36.106 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi173/crash 2024-06-06T03:21:36.106 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash -- . 2024-06-06T03:21:36.159 INFO:teuthology.orchestra.run.smithi173.stderr:tar: /var/lib/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-06-06T03:21:36.159 INFO:teuthology.orchestra.run.smithi173.stderr:tar: Error is not recoverable: exiting now 2024-06-06T03:21:36.160 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-06T03:21:36.160 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T03:21:36.194 INFO:teuthology.orchestra.run.smithi153.stdout:1717643432.5333953 mon.smithi153 (mon.0) 121 : cluster [WRN] Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-06T03:21:36.194 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-06-06T03:21:36.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[SEC\]' /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T03:21:36.267 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]' /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-06-06T03:21:36.338 INFO:teuthology.orchestra.run.smithi153.stdout:1717643491.6626096 mgr.smithi173.zwtggy (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents 2024-06-06T03:21:36.339 INFO:tasks.cephadm:Compressing logs... 2024-06-06T03:21:36.339 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T03:21:36.382 DEBUG:teuthology.orchestra.run.smithi173:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-06T03:21:36.404 INFO:teuthology.orchestra.run.smithi153.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T03:21:36.404 INFO:teuthology.orchestra.run.smithi153.stderr:: No such file or directory 2024-06-06T03:21:36.404 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T03:21:36.404 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-06-06T03:21:36.405 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T03:21:36.405 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi153.gidtue.log 2024-06-06T03:21:36.406 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-06-06T03:21:36.406 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log: /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi153.gidtue.log: 87.6% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T03:21:36.407 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-06-06T03:21:36.407 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T03:21:36.409 INFO:teuthology.orchestra.run.smithi173.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-06-06T03:21:36.409 INFO:teuthology.orchestra.run.smithi173.stderr:: No such file or directory 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip/var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.5.log: 92.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr: 87.8% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.log.gz 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.4.log: 90.4% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi173.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi173.zwtggy.log 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log: /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi173.log: 81.8% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T03:21:36.410 INFO:teuthology.orchestra.run.smithi173.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T03:21:36.411 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log 2024-06-06T03:21:36.411 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log: /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log: 83.4% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-06-06T03:21:36.412 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-06-06T03:21:36.412 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi153.log 2024-06-06T03:21:36.412 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi153.log: 93.7% -- replaced with /var/log/ceph/cephadm.log.gz 2024-06-06T03:21:36.415 INFO:teuthology.orchestra.run.smithi173.stderr:/var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi173.zwtggy.log: /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi173.zwtggy.log.gz 2024-06-06T03:21:36.417 INFO:teuthology.orchestra.run.smithi153.stderr: 90.9% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T03:21:36.419 INFO:teuthology.orchestra.run.smithi173.stderr: 90.8% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-06-06T03:21:36.452 INFO:teuthology.orchestra.run.smithi173.stderr: 93.0% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-06-06T03:21:36.457 INFO:teuthology.orchestra.run.smithi173.stderr: 93.0% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-06-06T03:21:36.464 INFO:teuthology.orchestra.run.smithi153.stderr: 93.1% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-06-06T03:21:36.466 INFO:teuthology.orchestra.run.smithi173.stderr: 92.5% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-06-06T03:21:36.490 INFO:teuthology.orchestra.run.smithi173.stderr: 92.9% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi173.log.gz 2024-06-06T03:21:36.492 INFO:teuthology.orchestra.run.smithi173.stderr: 2024-06-06T03:21:36.492 INFO:teuthology.orchestra.run.smithi173.stderr:real 0m0.096s 2024-06-06T03:21:36.492 INFO:teuthology.orchestra.run.smithi173.stderr:user 0m0.256s 2024-06-06T03:21:36.493 INFO:teuthology.orchestra.run.smithi173.stderr:sys 0m0.024s 2024-06-06T03:21:36.495 INFO:teuthology.orchestra.run.smithi153.stderr: 92.7% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-06-06T03:21:36.500 INFO:teuthology.orchestra.run.smithi153.stderr: 89.4% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mgr.smithi153.gidtue.log.gz 2024-06-06T03:21:36.520 INFO:teuthology.orchestra.run.smithi153.stderr: 93.3% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-06-06T03:21:36.673 INFO:teuthology.orchestra.run.smithi153.stderr: 90.6% -- replaced with /var/log/ceph/0b80117e-23b2-11ef-bc9b-c7b262605968/ceph-mon.smithi153.log.gz 2024-06-06T03:21:36.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-06-06T03:21:36.675 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m0.281s 2024-06-06T03:21:36.675 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m0.627s 2024-06-06T03:21:36.675 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.040s 2024-06-06T03:21:36.675 INFO:tasks.cephadm:Archiving logs... 2024-06-06T03:21:36.676 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi153/log 2024-06-06T03:21:36.677 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T03:21:36.864 DEBUG:teuthology.misc:Transferring archived files from smithi173:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi173/log 2024-06-06T03:21:36.865 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-06T03:21:36.947 INFO:tasks.cephadm:Removing cluster... 2024-06-06T03:21:36.948 DEBUG:teuthology.orchestra.run.smithi153:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 --force 2024-06-06T03:21:37.415 DEBUG:teuthology.orchestra.run.smithi173:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 0b80117e-23b2-11ef-bc9b-c7b262605968 --force 2024-06-06T03:21:37.888 INFO:tasks.cephadm:Removing cephadm ... 2024-06-06T03:21:37.889 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T03:21:37.906 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-06T03:21:37.922 INFO:tasks.cephadm:Teardown complete 2024-06-06T03:21:37.922 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T03:21:38.111 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e10594aa6fe5469898ffcc433160f93c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/cephadm.py", line 2237, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph.py", line 1478, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/git.ceph.com_ceph_3d746ca5346f34eef7e826a1eb71e59e02b682ae/qa/tasks/ceph_manager.py", line 3204, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-06T03:21:38.113 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-06T03:21:38.124 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-06T03:21:38.137 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-06T03:21:38.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T03:21:38.153 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-06T03:21:38.226 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-06T03:21:38.235 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-06T03:21:38.236 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T03:21:38.238 DEBUG:teuthology.orchestra.run.smithi173:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-06T03:21:38.251 INFO:teuthology.orchestra.run.smithi153.stderr:bash: line 1: ntpq: command not found 2024-06-06T03:21:38.252 INFO:teuthology.orchestra.run.smithi173.stderr:bash: line 1: ntpq: command not found 2024-06-06T03:21:38.255 INFO:teuthology.orchestra.run.smithi153.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T03:21:38.256 INFO:teuthology.orchestra.run.smithi153.stdout:=============================================================================== 2024-06-06T03:21:38.256 INFO:teuthology.orchestra.run.smithi153.stdout:^* hv01.front.sepia.ceph.com 3 6 377 13 +583us[ +588us] +/- 68ms 2024-06-06T03:21:38.256 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -860us[ -855us] +/- 80ms 2024-06-06T03:21:38.256 INFO:teuthology.orchestra.run.smithi153.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +197us[ +202us] +/- 86ms 2024-06-06T03:21:38.256 INFO:teuthology.orchestra.run.smithi153.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:21:38.257 INFO:teuthology.orchestra.run.smithi173.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-06T03:21:38.258 INFO:teuthology.orchestra.run.smithi173.stdout:=============================================================================== 2024-06-06T03:21:38.258 INFO:teuthology.orchestra.run.smithi173.stdout:^* hv01.front.sepia.ceph.com 3 6 377 12 +577us[ +597us] +/- 68ms 2024-06-06T03:21:38.258 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 12 -840us[ -819us] +/- 80ms 2024-06-06T03:21:38.258 INFO:teuthology.orchestra.run.smithi173.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 +193us[ +213us] +/- 86ms 2024-06-06T03:21:38.258 INFO:teuthology.orchestra.run.smithi173.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-06T03:21:38.258 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-06T03:21:38.267 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-06T03:21:38.267 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-06T03:21:38.282 DEBUG:teuthology.orchestra.run.smithi153:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T03:21:38.370 DEBUG:teuthology.orchestra.run.smithi173:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-06T03:21:38.435 DEBUG:teuthology.orchestra.run.smithi153:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T03:21:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:21:38.463 DEBUG:teuthology.orchestra.run.smithi173:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-06T03:21:38.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:21:38.491 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-06T03:21:38.501 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-06T03:21:38.509 INFO:teuthology.task.internal:Duration was 1186.204021 seconds 2024-06-06T03:21:38.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-06T03:21:38.518 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-06T03:21:38.518 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T03:21:38.521 DEBUG:teuthology.orchestra.run.smithi173:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-06T03:21:38.557 INFO:teuthology.orchestra.run.smithi153.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T03:21:38.571 INFO:teuthology.orchestra.run.smithi173.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-06T03:21:38.875 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-06T03:21:38.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-06-06T03:21:38.875 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T03:21:38.895 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi173.front.sepia.ceph.com 2024-06-06T03:21:38.896 DEBUG:teuthology.orchestra.run.smithi173:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-06T03:21:38.941 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-06T03:21:38.941 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T03:21:38.944 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-06T03:21:39.008 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-06T03:21:39.008 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T03:21:39.010 DEBUG:teuthology.orchestra.run.smithi173:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-06T03:21:39.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-06T03:21:39.409 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-06T03:21:39.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T03:21:39.434 DEBUG:teuthology.orchestra.run.smithi173:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-06T03:21:39.460 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-06T03:21:39.470 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T03:21:39.477 DEBUG:teuthology.orchestra.run.smithi173:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-06T03:21:39.498 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-06-06T03:21:39.523 INFO:teuthology.orchestra.run.smithi173.stdout:kernel.core_pattern = core 2024-06-06T03:21:39.561 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T03:21:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:21:39.585 DEBUG:teuthology.orchestra.run.smithi173:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-06T03:21:39.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-06T03:21:39.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-06T03:21:39.625 INFO:teuthology.task.internal:Transferring archived files... 2024-06-06T03:21:39.626 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi153 2024-06-06T03:21:39.626 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T03:21:39.670 DEBUG:teuthology.misc:Transferring archived files from smithi173:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-05_21:24:16-fs-squid-distro-default-smithi/7743004/remote/smithi173 2024-06-06T03:21:39.671 DEBUG:teuthology.orchestra.run.smithi173:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-06T03:21:39.706 INFO:teuthology.task.internal:Removing archive directory... 2024-06-06T03:21:39.706 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T03:21:39.709 DEBUG:teuthology.orchestra.run.smithi173:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-06T03:21:39.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-06T03:21:39.767 INFO:teuthology.task.internal:Not uploading archives. 2024-06-06T03:21:39.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-06T03:21:39.793 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-06T03:21:39.793 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T03:21:39.795 DEBUG:teuthology.orchestra.run.smithi173:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-06T03:21:39.807 INFO:teuthology.orchestra.run.smithi153.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 03:21 /home/ubuntu/cephtest 2024-06-06T03:21:39.810 INFO:teuthology.orchestra.run.smithi173.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 6 03:21 /home/ubuntu/cephtest 2024-06-06T03:21:39.812 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-06T03:21:39.821 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-06T03:21:39.925 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mgr mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn pg_health syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1186.204020500183 failure_reason: '"1717643491.6626096 mgr.smithi173.zwtggy (mgr.14230) 1 : cluster [ERR] Failed to load ceph-mgr modules: k8sevents" in cluster log' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e10594aa6fe5469898ffcc433160f93c status: fail success: false 2024-06-06T03:21:39.926 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-06T03:21:40.011 INFO:teuthology.run:FAIL